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Abstract 

In  the  past  20  years,  over  150  recommendations  have  been  made  to  improve 
software  systems  development  by  organizations  such  as  the  Defense  Science  Board, 
National  Research  Council  and  the  U.S.  General  Accountability  Office.  It  has  been 
discovered  that  many  of  these  recommendation  have  remained  unimplemented.  This 
research  had  the  purpose  of  confirming  the  application  of  these  previous 
recommendations  to  improve  software  acquisition  in  the  Aeronautical  Systems  Center. 
This  was  accomplished  through  interviews  with  20  software  practitioners  in  the 
acquisition  community  and  the  review  of  relevant  literature.  Through  the  analysis  of  the 
interviews  and  literature,  this  research  was  able  to  confirm  that  many  of  the 
recommendations  have  been  applied  in  programs  throughout  ASC. 
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SOFTWARE  ACQUISTION  IMPROVEMENT  IN  THE 


AERONAUTICAL  SYSTEMS  CENTER 


I.  Introduction 


Background 

United  States  military  operations  around  the  world  continue  to  show  the 
dominance  of  the  weapon  systems  being  developed  by  the  Department  of  Defense  (DoD) 
(Parcchia:  2004).  Though  the  DoD  continues  to  produce  dominant  systems,  the 
programs  that  develop  them  are  continually  plagued  with  cost  overruns  and 
unsatisfactory  performance  (Parcchia:  2004).  The  dependence  of  systems  on  software 
being  developed  has  continued  to  increase.  This  growth  of  software  dependence  can  be 
seen  when  looking  at  the  number  of  functions  perfonned  by  software  in  aircraft  over  the 
years  (Table  1).  In  1960,  the  F-4  Phantom  had  only  8%  of  it  functions  performed  by 
software.  Fifteen  years  later  the  F-15  Eagle  relied  on  software  for  35%  of  it  functions. 
This  number  continue  to  grow  over  the  years  and  the  Air  Force’s  newest  aircraft  the  F/A- 
22  Raptor,  relies  on  software  for  80%  of  its  functions  (GAO:  2000). 

Table  1.  Increasing  Software  Functions  (GAO:  2000) 


Aircraft 

Year 

%  Functions 
Performed  by 
software 

F-4 

1960 

8 

A-7 

1964 

10 

F-lll 

1970 

20 

F-15 

1975 

35 

F-16 

1982 

45 

B-2 

1990 

65 

F/A-22 

2000 

80 

1 


The  increase  in  functions  controlled  by  software  required  a  similar  increase  in 
Source  Lines  of  Code  (SLOC).  While  the  F-15A  had  only  60  thousand  lines  of  code,  as 
of  2004  the  F/A-22  had  2.1  million  lines  of  code  and  that  number  was  projected  to  grow 
(Dobornski:  2005).  Though  this  study  focused  on  aircraft  it  begins  to  show  how 
software  has  continued  to  grow  as  an  integral  part  of  systems  development. 

The  DoD’s  growing  dependence  on  software  systems  has  been  paralleled  by  an 
increase  in  the  number  of  problems  associated  with  the  acquisition  of  these  systems. 
According  to  a  1999  study  by  the  Standish  Group  only  16.2%  of  large  scale  government 
and  commercial  software  systems  were  completed  on  budget  and  schedule  (Linberg: 
1999).  These  programs  are  what  the  Standish  Group  considers  a  “project  success,”  while 
programs  that  deliver  behind  schedule  or  over  budget  were  considered  “project  failures” 
(Linberg:  1999).  Of  the  programs  studied  by  the  group,  52.7%  were  classified  project 
failures  with  the  remaining  31.1%  of  the  projects  cancelled  before  completion  (Linberg: 
1999). 

A  1998  study  by  Software  Productivity  Research,  Inc.  showed  that  as  the  amount 
of  software  grew  in  programs,  the  programs  were  unlikely  to  complete  within  budget  and 
schedule  (Jones:  1998).  There  was  even  a  greater  chance  of  cancellation  of  the  program 
when  the  amount  of  software  grew  to  a  size  of  what  was  considered  major  military 
systems  (Jones:  1998).  Jones  concluded  that  late  delivery,  increased  budget,  or 
cancellation  for  military  programs  often  occurred  due  to  lower  productively.  This 
decreased  level  of  productivity  in  the  production  of  DoD  software  was  primarily  due  to 
the  increased  standards  and  over  regulation  (Jones:  1998). 
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Instead  of  spending  time  writing  code,  developers  were  spending  time  completing  the 
paper  work  required  of  DoD  programs.  According  to  Jones  the  amount  of  paperwork 
required  was  almost  three  times  that  of  the  commercial  market. 

Table  2  shows  some  of  the  common  reasons  why  both  commercial  and  DoD 
software  programs  fail.  The  reasons  are  not  due  to  the  complexity  of  software 
development  but  due  to  the  management  of  the  processes  used.  The  Defense  Science 
Board  in  1987  wrote  in  a  report  on  military  software  that,  “The  task  force  is  convinced 
that  today’s  major  problems  with  military  software  development  are  not  technical 
problems,  but  management  problems.” 


Table  2.  Reasons  for  Project  Failure  (Charette:  2005) 


Unrealistic  or  unarticulated  project 
goals 

Poor  communication  among  developers  and 
users 

Inaccurate  estimates  of  resources 

Use  of  immature  technology 

Badly  defined  systems  requirements 

Inability  to  handle  project's  complexity 

Poor  reporting  of  the  project  status 

Sloppy  development  practices 

Unmanaged  risks 

Poor  project  management 

Stakeholder  politics 

Commercial  pressure 

Independent  government  organizations  such  as  the  Defense  Science  Board  (DSB), 
Government  Accounting  Office  (GAO),  and  National  Research  Council  (NRC)  have 
published  over  150  recommendations  to  improve  the  development  and  acquisition  of 
military  software.  Many  of  the  recommendations  have  remained  unimplemented  (GAO: 
2000).  This  means  that  the  potential  benefits  of  these  recommendations  have  yet  to  be 
seen.  Even  as  recent  as  September  of  2006  the  National  Defense  Industrial  Association 
(NDIA)  has  raised  concerns  regarding  software  development  in  the  military. 
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Purpose 

The  purpose  of  this  research  is  to  confirm  the  application  of  previous 
recommendations  to  improve  software  acquisition  in  the  Aeronautical  Systems  Center 
(ASC)  and  to  investigate  any  perceived  benefits.  This  study  will  be  accomplished  with 
the  review  of  literature  related  to  the  management  of  software  development  as  well  as 
previous  reports  by  the  GAO,  DSB,  and  NRC.  The  reports  have  been  shown  to  contain 
over  150  recommendations  to  improve  software  acquisition.  A  list  of  these 
recommendations  which  can  be  found  in  Appendix  B  will  be  utilized  to  formulate  a  set  of 
interview  questions  that  will  be  presented  to  practitioners  in  the  field.  The  results  of  the 
interviews  will  then  be  used  to  make  conclusions  for  the  purpose  of  this  study. 

Preview 

Chapter  two  of  this  document  will  present  a  detailed  summary  of  the  problem, 
relevant  research  and  past  software  recommendations.  Chapter  three  will  then  outline  the 
methodology  used  to  collect  and  analyze  relevant  data.  Chapter  four  presents  the  results 
of  the  data  collected  for  this  study.  Chapter  five  summarizes  the  results,  present 
conclusions,  and  provides  recommendations  for  future  related  work. 
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II.  Literature  Review 


Overview 

The  focus  of  this  research  is  based  on  recommendations  to  improve  software 
development  and  acquisition  made  by  the  DSB,  NRC,  and  GAO.  These  reports  raise 
issues  common  to  many  of  today’s  programs.  This  chapter  will  present  relevant  issues 
raised  by  the  DSB,  NRC,  and  GAO  which  affect  the  DoD.  Along  with  previous  reports, 
this  chapter  will  also  discuss  relevant  literature  related  to  software  development. 
Problems  facing  Acquisition  Regulation  /Process  for  Software  Development 

When  developing  software  a  developmental  model  is  often  used.  In  the  early 
1980s  the  model  used  by  the  DoD  was  the  waterfall  model,  which  was  mandated  by 
DoD-STD-2167  (DSB:  1987).  The  waterfall  model  assumes  a  non-iterative  development 
process  (NRC:  1989)  in  which  development  efforts  sequentially  flow  from  the  first  step 
to  the  last  step  and  only  one  step  at  a  time,  as  shown  in  Figure  1.  The  only  potential  for 
iteration  is  with  the  feedback  loops  shown  in  the  figure. 


Figure  1.  Waterfall  Model  (Overmyer:  1990) 
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These  feedback  loops  are  often  not  used  due  to  a  large  amount  of  documentation 
to  complete  the  process  (Overmyer:  1990).  System  requirements  were  also  determined 
up-front  and  user/developer  interaction  occurred  only  at  the  beginning  of  the  program 
(DSB:  1987).  According  to  the  NRC,  the  waterfall  model  was  characterized  by 
unrealistic  specifications  and  was  measured  by  documentation  not  by  demonstrable 
results  (NRC:  1989).  This  meant  that  success  of  the  project  was  not  known  until 
completion.  To  solve  these  issues  it  was  recommended  that  the  DoD  use  a  more  iterative 
model.  The  model  recommended  should  have  multiple  occasions  of  interactions  between 
the  user  and  developer,  evolving  requirements,  and  demonstrable  prototypes  throughout 
the  process  (DSB:  1987,  NRC:  1989) 

Not  only  was  the  waterfall  model  being  imposed  on  programs,  but  also  other 
unnecessary  standards  and  specifications  (NRC:  1989).  In  1983,  there  was  a  mandate  to 
use  the  Ada  programming  language.  The  intent  was  to  create  a  common  programming 
langue  for  DoD  systems  (Brosgol:  2001).  The  1987  DSB  report  agreed  with  the  push  for 
a  standard  programming  langue  and  recommended  more  emphasis  be  placed  on  the 
management  of  the  Ada  language.  In  1983  a  mandate  was  created  for  the  required  use  of 
Ada  (Brosgol:  2001).  This  mandate  lasted  until  1997,  when  it  was  realized  that  the 
commercial  market  was  not  supporting  Ada  (Brosgol:  2001). 

The  DoD  depended  on  the  commercial  market  for  the  development  of  compiler 
and  tool  development  for  Ada  programming  (Reifer:  2000).  However,  the  commercial 
market  has  been  marketing  more  popular  languages  such  as  C  and  C++  (Brosgol:  2001). 
With  less  emphasis  on  Ada’s  use  and  tool  development  it  has  been  reported  that  only  one 
in  ten  defense  systems  are  using  Ada  for  software  development  (Reifer:  2000). 
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Mandates  such  as  the  use  of  the  waterfall  model  and  Ada  language  prompted  the 
NRC  to  recommend  that  programs  be  allowed  to  tailor  their  processes  to  the 
characteristics  of  their  programs.  This  would  encourage  program  manager  to  be  more 
innovative  and  less  restricted  by  standards  and  specifications  (NRC:  1989).  The  DSB 
also  recommended  tailoring  of  the  process  based  on  certain  classifications.  These 
classifications  included,  life  cycle  model,  acquisition  strategy,  requirement  stability, 
reuse  potential,  contract  support  strategy,  and  evaluation  strategy  (DSB:  1989).  Along 
with  tailoring  of  systems  risk  reduction  actives  should  also  occur,  including  prototyping 
hardware  and  tracing  designs  back  to  user  requirements  (DSB:  1989). 

An  important  aspect  of  requirements  is  that  they  be  feasible;  unfeasible 
requirements  can  negatively  impact  effectiveness,  cost,  and  schedule  (DSB:  1987).  To 
ensure  that  the  requirements  are  feasible,  it  was  recommended  that  there  be  a  mutual 
understanding  between  the  contractor,  program  office,  and  user  (GAO:  2004a).  In  order 
to  achieve  this  understanding,  knowledge  should  be  gained  from  performing  systems 
engineering  analysis  prior  to  the  start  of  development  (GAO:  2004a).  After  development 
has  started  it  is  still  critical  to  review  changes  to  requirements. 

In  2006  the  NDIA  recognized  that  impacts  of  changes  to  requirements  were  still 
not  being  consistently  addressed  by  program  managers  (NDIA:  2006).  Like  the  GAO  in 
2004,  the  NDIA  in  2006  suggested  that  an  assessment  of  changed  requirements  and  the 
impacts  be  conducted.  The  trade-off  analysis  of  the  requirements  should  consider  the 
impacts  on  cost,  schedule,  and  perfonnance  (GAO:  2004a).  Once  a  trade-off  analysis  has 
been  completed,  it  was  recommended  that  program  mangers  be  given  the  authority  to 


7 


decide  the  proper  time  to  begin  the  development  of  new  or  changed  requirements  (NRC: 
1989). 

To  track  progress  and  ensure  that  developers  are  meeting  cost,  schedule,  and 
performance  measures,  it  was  recommended  that  metrics  be  provided  to  the  program 
offices  (GAO:  2004a).  The  metrics  should  include  cost,  schedule,  size,  requirements, 
test,  defects,  and  quality  (GAO:  2004a).  The  metrics  will  “portray  variances  between 
planned  and  actual  performance”  and  should  provide  for  early  detection  of  potential 
problems  (Sambur:  2004). 

In  2002  the  GAO  recommended  that  the  Defense  Logistics  Agency  (DLA) 
implement  a  software  process  improvement  program  due  to  a  lack  of  a  mature  software 
acquisition  process  (GAO:  2002).  The  GAO  recommended  Carnegie  Mellon 
University’s  Software  Engineering  Institute  (SEI)  Software  Acquisition  Capability 
Maturity  Model  (SA-CMM)  (GAO:  2002).  This  model  is  used  to  determine  the  maturity 
of  software  development  process  in  an  organization.  The  model  contains  five  levels  in 
which  an  organization  can  be  categorized.  The  first  level  means  that  an  organization  has 
ad  hoc  and  ill  defined  processes  (GAO:  2002).  As  the  organization  progresses  to  level 
five,  the  process  become  more  organized,  repeatable  and  transferable  throughout  the 
organization  (GAO:  2002).  Once  an  organization  reaches  level  five,  processes  are 
institutionalized  and  the  organization  can  focus  on  continuous  improvement  (GAO: 
2002). 

The  GAO  found  that  the  DLA  was  not  meeting  level  two  requirements  on  key 
process  areas  and  was  not  at  level  three  on  risk  management.  The  GAO  recommended 
that  a  policy  be  established  stating  that  the  DLA  attain  at  least  these  SA-CMM  levels. 


The  DoD  recognized  the  need  for  software  process  improvement  plans  and  in 
2003  these  plans  were  required  to  be  implemented  DoD  wide  by  Section  804  of  the  2003 
National  Defense  Authorization  Act  (GAO:  2004a).  This  section  directs  the  secretary  of 
each  military  agency  to  develop  programs  to  improve  their  software  acquisition  process 
(DSB:  2000).  The  improvement  programs  are  required  to  have  a  documented  process  for 
planning,  requirements  development/management,  project  management/oversight,  and 
risk  management  (DSB:  2000).  The  program  must  also  include  a  plan  for  producing  the 
appropriate  metrics  (DSB:  2000).  The  metrics  will  be  used  to  measure  the  performance 
of  the  programs  and  to  help  in  continuous  process  improvement  (DSB:  2000). 

Use  of  Commercial-Off-the-Shelf  Products  in  the  Department  of  Defense 

Commercial-of-the-Shelf  (COTS)  software  products  have  been  shown  to  have 
both  pros  and  cons  in  DoD  systems  development.  Since  COTS  products  are  available 
early  in  the  design  phase,  systems  can  be  designed  to  incorporate  COTS  products  which 
can  save  time  and  money  in  systems  development  (DSB:  1994).  Another  positive  aspect 
is  in  the  support  and  operation  of  COTS  products.  Many  of  the  products  come  with 
documentation  and  training  material  along  with  the  availability  of  already  trained 
personnel  from  the  commercial  market  (Anderson:  1998).  With  the  commercial  market 
continually  advancing  technology  the  DoD  can  then  leverage  the  commercial  market  for 
the  most  advanced  products  (Anderson:  1998). 

Though  the  DoD  has  become  more  accepting  of  COTS  software  products  there 
are  still  risks  associated  with  their  use  in  defense  systems.  COTS  products  offer  a  variety 
of  security  risks  including  software  assurance.  Software  assurance  is  concerned  with  the 
security  risks  of  the  software.  The  risks  can  include  malicious  code  that  is  written  into 
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the  COTS  products  that  are  developed  by  foreign  countries.  The  use  of  foreign 
developers  is  at  the  discretion  of  program  managers  (GAO:  2004b).  With  the  threat  of 
malicious  code  there  is  an  increased  need  in  to  test  COTS  products  being  used.  The 
NDIA  suggest  that  with  the  increase  of  required  testing,  the  DoD  should  look  at 
reviewing  the  testing  processes  to  include  a  reduction  in  the  required  documentation  and 
an  increase  in  training  of  testing  personnel  to  ensure  products  are  safe  for  use  in  DoD 
systems  (NDIA:  2006). 

Even  with  the  risks  associated  with  COTS  products,  it  has  been  recommended 
that  the  DoD  take  advantage  of  commercial  products  (DSB:  1994).  It  was  suggested  that 
the  DoD  look  to  the  commercial  market  to  buy  tools,  methods,  environments,  and 
application  software,  instead  of  custom-built  software  (DSB:  1987).  These  products 
should  only  be  considered  if  trade-off  studies  and  analysis  of  potential  reuse  of  existing 
COTS  products  have  been  accomplished  (GAO:  1994).  The  NDIA  in  2006 
recommended  that  these  trade-off  studies  be  reviewed  at  each  milestone  and  major 
reviews  to  ensure  lifecycle  cost  are  continually  being  addressed. 

Increasing  Research  and  Development  of  Software  Systems 

It  has  been  suggested  that  the  commercial  market  is  leading  today’s  information 
technology  (DSB:  2000).  However,  the  commercial  market  can  not  cover  all  the  areas 
needed  for  unique  military  systems  (DSB:  2000).  The  technologies  needed  for  the 
military  systems  are  continually  changing  due  to  new  operational  threats  and  rapidly 
changing  requirements  (NRC:  1989).  It  was  recommended  that  the  DoD  fund  technology 
programs  to  meet  the  unique  demands  of  DoD  systems  (NRC:  1989). 


10 


Problems  facing  Software  Personnel 

In  1987  it  was  stated  that  the  DoD  should  assume  that  it  will  not  be  getting  any 
more  personnel  in  the  software  field  and  therefore  should  plan  how  to  best  use  current 
personnel  (DSB:  1987).  In  2006,  a  similar  statement  was  made  by  the  NDIA  which 
stated  that  there  is  an  insufficient  quantity  and  quality  of  software  engineers  to  meet  the 
demands  of  the  government.  Reasons  given  by  the  NDIA  for  the  lack  of  software 
engineers  were  insufficient  career  incentives,  competition,  and  inadequate  funding 
(NDIA:  2006).  To  combat  these  issues  it  was  recommended  that  the  DoD  improve  the 
education  and  career  field  of  its  available  personnel  (NDIA:  2006,  DSB:  1987).  This 
would  include  prior  to  program  initiation  and  at  appropriate  intervals,  program  managers 
should  receive  software-intensive  systems  training  (DSB:  2000).  To  further  educate 
software  personnel  it  was  suggested  that  a  graduate-level  program  for  software  engineers 
be  created  (DSB:  2000).  It  was  also  recommended  that  government/contractor  teams 
receive  team  and  software  refresher  training  (DSB:  2000).  This  idea  of  refresher  training 
along  the  rotation  of  government/contractor  personnel  between  the  program  office  and 
the  developing  organization  was  intended  to  foster  team  building  and  sharing  of 
knowledge  (DSB:  1994). 
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III.  Methodology 


Overview 

The  purpose  of  this  research  is  to  confirm  the  application  of  previous 
recommendations  to  improve  software  acquisition  in  ASC  and  to  investigate  any 
perceived  benefits.  To  accomplish  this  study,  practitioners  in  the  software  acquisition 
field  will  be  interviewed.  The  data  gained  from  the  interviews  will  be  used  to  confirm 
whether  the  recommendations  have  been  applied  within  ASC.  The  first  decision  in 
formulating  the  research  methodology  is  to  decide  between  a  qualitative  and  quantitative 
study. 

Selection  of  Research  Method 

Quantitative  studies  require  the  use  of  standard  measures  so  that  many 
perspectives  and  experiences  can  fit  into  predetennined  responses  (Patton:  2002).  There 
are  advantages  to  a  quantitative  approach  giving  the  researcher  the  ability  to  make 
statistical  generalizations  on  a  larger  number  of  cases  (Patton:  2002).  While  ruling  out 
statistics,  qualitative  research  allows  the  researcher  the  ability  to  gain  more  depth  and 
detail  on  a  limited  number  of  cases  (Patton:  2002).  For  these  reasons  a  qualitative  study 
was  chosen.  In  order  to  complete  this  study  a  standard  set  of  interview  questions  was 
used.  However,  due  to  varying  experiences  and  the  desire  to  seek  the  more  in-depth 
experiences  of  interviewees  questions  were  left  open-ended  allowing  for  a  more  open 
study  of  the  research  area  and  the  understandings  and  imaginations  of  the  participants 
(Mason:  2002). 


12 


Patton  suggests  that  qualitative  data  can  be  gathered  in  three  ways:  in-depth  open- 
ended  interviews,  direct  observations,  or  through  written  documents  (Patton:  2002).  For 
this  research  open-ended  interviews  were  selected.  This  type  of  data  gathering  results  in 
“in-depth  responses  about  people’s  experiences,  perceptions,  opinions,  feelings,  and 
knowledge”  (Patton:  2002).  Qualitative  studies  have  also  been  known  to  provide  benefit 
in  understanding  organizational  goals,  processes,  and  failures  in  policies  (Skinner:  2000). 
The  type  of  data  collected  from  this  study  may  lead  future  research  to  conduct  a 
quantitative  study  on  the  same  subject  area. 

For  these  reasons,  it  was  determined  that  opened-ended  interviews  would  be  the 
method  used  for  data  collection.  Open-ended  questions  do  not  limit  interviewees  to 
alternatives  set  by  the  researcher  (Schuman:  1981).  Open-ended  questions  also  avoid 
imparting  suggestions  or  imposing  answers  that  the  interviewee  has  not  considered 
(Shuman:  1981). 

Interview  Questions  Development 

The  questions  were  developed  with  the  purpose  of  confirming  whether  the 
programs  at  the  Aeronautical  Systems  Center  had  applied  the  recommendations  to 
improve  software  acquisition  and  any  perceived  benefits  to  their  programs.  A  list  of 
recommendations  was  compiled  from  independent  government  agencies  that  have  been 
tasked  by  the  DoD  and  Congress  to  investigate  software  development  and  acquisition 
within  the  DoD. 

Among  the  more  than  150  recommendations  there  were  many  duplicates  and 
several  pertained  to  programs  and  organizations  that  were  no  longer  in  existence. 

Through  research  and  expert  opinion,  recommendations  that  were  determined  no  longer 
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relevant  were  omitted  from  interview  questions.  For  example,  recommendations  that 
focused  on  the  use  of  the  Ada  programming  language  were  omitted  because  Ada  is  no 
longer  mandated  by  the  DoD. 

Throughout  the  review  process  each  recommendation  was  placed  into  categories 
based  on  the  context  of  the  recommendation.  The  categories  were  created  due  to  the 
numerous  recommendations  and  to  make  them  more  manageable.  Many 
recommendations  may  fit  into  multiple  categories;  however  it  was  the  discretion  the 
researcher  to  place  them  into  the  categories  selected.  This  allowed  for  a  better  flow 
during  interview  question  development.  Much  iteration  took  place  until  the  remaining 
recommendations  were  categorized  as:  Policy,  Research  and  Development,  Best 
Practices,  Lifecycle,  Source  Selection,  COTS,  Project  Management,  Metrics,  Personnel, 
Test  and  Evaluation,  and  Support. 

Once  placed  into  the  above  categories  the  recommendations  were  reviewed  again. 
Similar  recommendations  were  restructured  and  combined  into  a  set  of  forty-six 
interview  questions.  The  forty-six  interview  questions  were  also  aligned  in  the  above 
categories  and  used  as  section  headings  on  the  interview  form.  This  gave  the  interview 
participant  a  context  for  each  set  of  questions.  This  form  can  be  found  in  Appendix  A. 
Selection  of  Research  Subjects 

Research  subjects  were  selected  based  on  their  knowledge  and  experience  in 
software  development  and  acquisition  in  DoD  programs.  The  scope  of  this  study  was 
limited  to  individuals  who  work  at  the  Aeronautical  Systems  Center  (ASC),  Wright- 
Paterson  Air  Force  Base  so  interviews  could  be  conducted  in  person  and  also  due  to  lack 
of  funding.  To  detennine  if  there  might  be  any  differences  in  those  individuals  who 
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worked  at  ASC  and  other  Air  Force  organizations  a  four  interview  participants  were 
selected  from  the  554th  Electronic  Systems  Group  (ELSG),  also  located  at  Wright 
Patterson  Air  Force  Base. 

In  order  to  locate  individuals  at  ASC  a  senior  engineering  leader  within  ASC  was 
consulted  to  provide  names  of  individuals  that  met  the  criteria  necessary  for  this  study.  A 
list  of  Chief  Engineers  from  fifteen  different  wings  within  ASC  was  also  obtained.  Those 
individuals  along  with  personal  contacts  of  the  researcher  were  contacted  and  asked  to 
provide  names  of  individuals  which  fit  the  criteria.  Individuals  at  the  554th  ELSG  were 
identified  in  a  similar  fashion.  A  list  of  names  of  potential  interview  participants  was 
developed  and  those  individuals  were  contacted  via  email  or  phone  and  asked  to 
participate  in  the  voluntary  interview.  At  the  conclusion  of  each  interview  the  participant 
was  asked  to  recommend  others  to  participate  in  this  research.  These  new  individuals 
were  then  contacted  and  interviews  continued. 

For  this  study  seventy-five  individuals  were  contacted.  Twenty-six  were  not 
considered  for  the  study,  but  used  to  gain  potential  candidates  to  interview.  The 
remaining  forty-nine  were  considered  potential  interview  participants  and  were  contacted 
to  be  interviewed.  For  various  reasons  some  individuals  declined  to  participant  in  this 
study.  Many  individuals  did  not  return  phone  calls  or  e-mails.  Others  had  already  moved 
to  different  positions  or  locations.  On  several  occasions  many  believed  they  were 
mistakenly  identified  by  leaders  as  good  candidates  since  they  have  not  worked  with 
software  in  at  least  five  years. 
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Ultimately,  twenty  interviews  were  conducted  in-person  at  the  individual’s 
location  and  each  lasted  approximately  forty-five  minutes.  Seventeen  individuals 
representing  thirteen  different  aircraft  programs  within  ASC  came  from  six  of  the  nine 
aircraft  product  wings.  Fourteen  worked  on  avionics  suites  and  other  embedded  software 
systems,  which  are  not  stand  alone  systems  but  are  integrated  into  an  overall  system. 

Two  other  participants  worked  on  simulator  systems  and  the  remaining  individual 
worked  with  an  optical  pod.  The  three  interviewed  at  the  554th  ELSG  worked  on  three 
different  business  systems. 

Experience  levels  of  participants  varied  for  both  acquisition  and  software 
experience  from  five  to  thirty-five  years.  The  years  of  experience  for  each  interview  can 
be  found  in  Table  3.  The  interviewees  had  an  average  of  19.7  years  of  acquisition 
experience.  Software  experience  varied  from  two  to  thirty-five  years  with  an  average  of 
15.6  years  of  experience.  At  the  time  of  the  interview  all  individuals  were  working  on 
software  portions  of  Air  Force  acquisition  programs. 

Table  3.  Years  of  Experience 


Interview 

Acquisition 

Software 

1 

25 

25 

2 

18 

18 

3 

23 

17 

4 

9 

2 

5 

8 

8 

6 

25 

25 

7 

17 

12 

8 

34 

30 

9 

32 

15 

10 

20 

20 

Interview 

Acquisition 

Software 

11 

5 

5 

12 

18 

4 

13 

7 

7 

14 

10 

1.5 

15 

26 

19 

16 

24 

24 

17 

23 

23 

18 

21 

8 

19 

35 

35 

20 

13 

13 
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Data  Analysis  Process 

Patton  recommended  that  in  order  to  analyze  the  data  collected  from  interviews, 
the  data  should  be  compiled  into  a  readable  narrative,  with  major  themes  and  identified 
categories  (Patton:  2002).  These  “themes,  patterns,  understandings,  and  insight  that 
emerge  from  fieldwork  and  subsequent  analysis  are  the  fruit  of  qualitative  research” 
(Patton:  2002).  The  purpose  of  qualitative  analysis  is  to  connect,  describe  and  classify 
the  data  collected  (Dey:  1993).  For  this  study  answers  were  synthesized  into  a  narrative 
that  presents  both  approving  and  dissenting  opinions  of  the  interviewees. 

To  accomplish  the  data  analysis,  answers  given  by  the  interviews  were  compiled 
into  a  spreadsheet  to  allow  for  comparison.  This  spreadsheet  can  be  found  in  Appendix 
D  of  this  document.  Similar  answers  were  grouped  together  and  themes  and  patterns 
were  acknowledged  and  reported  in  chapter  four,  which  shows  the  results  of  the  study. 
The  following  is  an  example  of  how  this  researcher  took  the  information  gathered  and 
identified  themes  and  patterns  providing  results  and  conclusions.  The  questions 
concerning  best  practices  asked  in  the  questionnaire  are,  ‘In  your  experience  has  the  DoD 
been  effective  at  collecting  and  disseminating  best  practices  of  both  the  government  and 
industry?’  The  other  question  is  “Is  there  a  process  to  evaluate  the  usage  of  best 
practices?’  After  conducting  the  twenty  interviews,  the  researcher  created  a  spreadsheet 
consolidating  the  interviewers’  responses  and  comments  about  best  practices.  This 
allowed  for  easier  analysis.  According  to  the  results  from  the  interviews,  the  majority  of 
respondents  thought  that  the  government  is  effective  at  collecting,  but  not  necessarily  at 
disseminating  best  practices. 
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Also,  most  said  there  is  no  process  to  evaluate  the  usage  of  best  practices.  Some 
of  the  comments  which  led  to  this  determination  were  as  follows:  ‘government  tries,  but 
typically  there  is  so  much  turnover  of  personnel  we  are  learning  lessons  over  and  over 
again,’  ‘not  totally  effective  in  best  practice  dissemination,  it  may  be  there,  but  [one]  is 
not  told  where  to  get  it,’  ‘they  didn’t  really  use  metrics,  but  they  looked  at  processes.” 
All  response  to  these  questions  can  be  found  in  Appendix  D.  The  results  and  analysis  for 
best  practices  is  a  combination  of  the  answers  from  the  above  two  questions.  Based  on 
these  results,  the  conclusion  can  be  made  that  the  government  should  create  programs  to 
increase  the  effectiveness  of  information  dissemination  of  best  practices.  Chapter  four 
provides  the  results  from  all  questions  and  respondents,  followed  by  chapter  five,  which 
draws  conclusions  and  recommendations  based  on  the  results  presented  in  chapter  four, 
such  as  the  example  just  described. 
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IV.  Results 


Overview 

This  chapter  will  provide  data  gained  from  the  interviews  described  in  chapter 
three  of  this  document.  The  individuals  interviewed  had  a  variety  of  experiences  in 
software  development  and  acquisitions.  Since  interviews  were  conducted  as  non¬ 
attribution,  specifics  tying  individuals  to  programs  have  been  omitted  for  confidentiality 
reasons.  Data  was  separated  into  categories  corresponding  to  the  questions  from  each 
group  based  on  the  methodology  described  in  chapter  three  of  the  document.  The  full 
questionnaire  used  during  interviews  can  be  found  in  Appendix  A. 

This  research  included  respondents  from  both  ASC  and  the  554th  ELSG  to 
determine  if  the  questions  asked  would  results  in  different  answers  between 
organizations.  No  significant  differences  were  observed  in  the  answers  given.  Therefore 
the  results  and  conclusions  will  be  discussed  with  no  distinctions  made  for  the  different 
organizations. 

Policy 

The  first  question  in  the  policy  section  focused  on  a  general  question  whether  or 
not  there  is  clear  set  of  acquisition  policy  and  more  specifically,  if  this  guidance  was  up- 
to-date  enough  to  accomplish  their  program.  In  both  cases,  three-fourths  of  the 
respondents  stated  there  is  an  adequate  amount  of  policy  and  up-to-date  guidance  to 
accomplish  the  program.  Rather  than  a  lack  of  software  policy,  six  subjects  suggested 
that  confusion  results  from  excessive  policy  that  cannot  easily  be  located.  To  further 
perplexity,  the  policy  and  guidance  was  evolving  and  constantly  changing. 
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The  subjects  were  also  asked  if  they  were  familiar  with  Section  804  of  the 
National  Defense  Authorization  Act,  its  implementation,  and  any  results  from  its 
implementation.  Only  two  individuals  had  specific  knowledge  of  Section  804  and  both 
agreed  it  was  providing  some  benefits.  All  interviewees  were  asked  to  read  an  excerpt  of 
Section  804  supplied  to  them  during  the  interview.  Afterwards,  five  individuals  stated 
that  they  were  not  aware  of  Section  804,  and  could  not  offer  more  information.  The 
remaining  participants  had  no  specific  knowledge  of  the  section,  but  after  looking  at  the 
excerpt  indicated  that  they  had  seen  some  impacts  to  their  program.  These  impacts  were 
not  always  a  direct  result  of  the  Section  804  recommendation;  but  results  of  polices  that 
were  similar  to  those  directed  by  Section  804. 

Research  and  Development 

The  next  section  of  the  interview  regarded  research  and  development  of  software 
technologies.  Interviewees  agreed  with  the  statement  that  the  commercial  market  is 
driving  the  information  technology  market.  It  was  suggested  that  there  are  areas  within 
the  commercial  market  not  covered  or  should  be  covered  by  the  DoD.  The  areas  include 
safety  critical  systems  and  security.  One  reason  given  to  focus  on  these  areas  is  the  DoD 
has  unique  requirements  in  which  the  commercial  market  may  not  be  interested.  Two 
individuals  also  suggested  that  their  programs  looked  to  the  commercial  market  for 
hardware,  but  not  for  software.  They  indicated  that  the  rest  of  the  DoD  should  do  the 
same.  This  impart  is  due  to  the  unique  software  applications  required  by  DoD  systems. 
When  discussing  going  to  the  civilian  market  for  tools,  methods,  environments,  and 
application  software,  all  participants  agreed  DoD  ought  to  continue  this  process.  It  was 
suggested  it  would  be  too  costly  to  produce  and  maintain  DoD  unique  software. 


20 


Best  Practice 


It  was  not  widely  agreed  that  both  the  government  and  industry  have  been 
effective  at  collecting  and  disseminating  best  practices.  One  suggested  the  government 
was  effective  at  collecting  the  practices,  but  not  necessarily  disseminating  those  practices 
to  other  organizations.  Another  observed  that  though  the  government  may  not  be 
effective,  it  is  getting  better  than  it  has  been  in  the  past.  Most  participants  were  not  aware 
of  a  process  to  evaluate  the  usage  of  best  practices.  Those  that  stated  there  was  a  process 
believed  it  was  the  responsibility  of  external  resources.  These  resources  could  be  from 
educational  sources.  One  of  the  sources  named  was  the  Air  Force  Institute  of  Technology 
which  is  a  graduate  school  for  the  Air  Force.  Also  named  was  the  Air  Force  Research 
Laboratory  which  provides  research  into  technologies  to  support  the  warfighter.  Finally, 
the  Acquisition  Center  of  Excellence  was  named.  This  center  provides  expert  advice  to 
those  in  the  acquisition  arena.  Non-educational  sources  could  be  the  ASC  engineering 
home  office  or  through  contractors  like  MITRE.  For  others  it  was  an  internal  process, 
lessons  were  learned  from  young  developers  or  by  comparing  present  developments  with 
past  developments. 

Lifecycle 

It  was  recommended  that  the  DoD  should  not  use  the  waterfall  model  and  instead 
use  the  Evolutionary  Acquisition  Process  (DSB:  1987).  All  interviewed  agreed  it  is 
appropriate  to  use  Evolutionary  Acquisition  for  software  development.  However,  not  all 
interviewees  agreed  it  should  be  the  primary  model.  Some  suggested  that  programs 
should  choose  the  model  that  works  best  for  their  program;  others  did  not  specify 
programs  having  the  option  to  choose  the  appropriate  model. 
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Interviewees  were  asked  if  programs  should  be  allowed  to  tailor  the  acquisition 
process  based  on  classifications  including:  life  cycle  model,  requirements  stability,  and 
the  use  of  COTS.  All  agreed  that  their  programs  should  be  allowed  to  tailor  the 
acquisition  process.  What  was  not  agreed  upon  was  the  extent  and  on  what 
classifications  the  process  should  be  tailored.  However,  when  asked  if  there  is  policy  or 
guidance  on  how  to  tailor  the  process  a  variety  of  answers  were  given.  A  majority 
responded  that  there  is  no  guidance.  Although  some  suggested  that  there  was  guidance,  it 
was  hard  for  individuals  to  name  a  specific  document  that  described  how  to  tailor  the 
process.  In  fact  no  specific  documents  were  given  at  the  time  of  the  interviews. 

Source  Selection 

A  source  selection  question  about  policy  requiring  the  government  and/or 
contractor  to  reach  a  particular  capability  maturity  level  resulted  in  many  varying 
answers.  Those  who  agreed  there  was  a  policy  could  not  identify  a  specific  policy  that 
required  obtaining  a  precise  maturity  level.  Regardless  of  whether  or  not  the 
interviewees  thought  there  was  a  policy,  it  was  widely  suggested  that  contractors  should 
obtain  at  least  CMMI  level  3. 

When  asked  if  evaluating  competitors  on  their  technical  approach  rather  than  cost 
was  feasible,  it  was  agreed  it  was  practical.  Technical  approach  was  considered  as 
important  as  cost  and  could  provide  the  best  value  approach.  One  of  the  interviewees 
stated  that  the  technical  approach  should  be  considered  a  trade-off  to  cost. 

Interviewees  were  asked  if  the  government  should  perform  an  analysis  of  COTS 
and  other  contractor  products  in  order  to  receive  a  best  value  solution.  Respondents 
agreed  that  this  recommendation  was  beneficial  and  often  accomplished  in  many 
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programs.  This  also  involved  an  analysis  of  contractor  past  performance  in  the 
integration  of  COTS  products.  It  was  suggested  that  the  analysis  of  the  products 
themselves  might  be  difficult  to  conduct  since  requirements  might  not  be  established  at 
this  phase  of  the  program.  This  same  reason  was  also  given  when  asked  if  contractors 
should  demonstrate  as  much  pre-existing  functionality.  It  could  be  hard  to  perform  since 
the  contractors  may  not  have  developed  programs  which  meet  the  new  requirements. 
COTS 

A  specific  system  could  not  be  identified  when  asked  if  there  was  one  to  help 
identify  potential  COTS  products.  Respondents  indicated  COTS  systems  could  be 
identified  through  various  avenues.  These  included  identification  by  the  contractor  and 
market  research  to  include  industry  conferences  or  internet  searches.  Others  may  have 
support  contractors  such  a  Gartner  Research  to  aid  their  program  in  seeking  the 
appropriate  products. 

Interviewees  agreed  that  program  managers  should  not  assume  that  software 
requirements  can  be  met  with  off-the-shelf  products.  It  was  suggested  that  a  thorough 
analysis  must  be  completed  to  consider  COTS.  Modification  of  COTS  products  was 
discussed  on  several  occasions.  It  can  be  costly  to  modify  COTS  products  and  therefore 
they  are  not  always  the  best  solution.  This  discussion  leads  into  the  next  question, 
whether  or  not  the  modification  of  commercial  products  should  be  discouraged. 

Two-thirds  of  the  interviewees  agreed  that  the  modification  of  COTS  products 
should  be  discouraged.  Once  modified,  the  product  can  be  more  costly  to  continue 
development  and  support  into  sustainment.  All  of  the  other  interviewees  agreed  that 
COTS  could  be  modified  to  meet  the  requirements. 
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Project  Management 

A  common  answer  for  program  management  was  received  when  asked  whether  it 
is  beneficial  for  program  managers  to  manage  price,  schedule,  and  functionality  but 
constrain  two  of  the  three.  The  answer  was  that  realistically  a  program  manager  must 
manage  all  three.  A  few  interviewees  thought  this  is  not  a  realistic  approach.  However, 
it  was  not  agreed  on  which  two  must  be  constrained. 

Interviewees  either  stated  that  one  never  really  knows  if  requirements  are  feasible 
or  they  described  various  reviews  and  documents  that  can  assist  in  the  determination  of 
feasibility.  This  carried  over  into  the  decision  to  determine  if  the  program  office  and 
contractor  have  the  same  understanding.  It  was  suggested  through  meetings,  integrated 
product  teams,  reviews,  and  documentation  a  mutual  understanding  can  be  accomplished. 

Performing  a  trade-off  analyses  for  major  changes  to  requirements  were  widely 
used  by  interviewees.  Two  individuals  suggested  that  the  analyses  are  conducted  at  the 
component  (hardware/software)  level  rather  than  solely  at  the  software  level.  Not  all 
discussed  the  level  at  which  the  analysis  was  performed.  Another  individual  stated 
analysis  may  not  be  feasible  due  to  demands  by  the  user  representative.  The  user 
representative  may  be  willing  to  assume  the  risk  to  push  for  the  product. 

It  was  recommended  that  program  managers  be  allowed  to  defer  late  requirements 
to  future  releases  (NRC:  1989).  Comments  on  this  concept  yielded  two  distinct 
responses.  First,  indicating  that  program  managers,  in  concert  with  user  representatives, 
should  have  the  authority  to  defer  requirements.  Alternatively,  some  individuals  stated 
that  requirements  deferral  should  be  the  purview  of  user  representatives  only. 
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Software  architectures  were  not  widely  perceived  as  beneficial  by  interviewees. 

In  many  cases  there  was  no  knowledge  of  architecture  and  how  it  was  used.  It  was  said 
that  architectures  in  place  were  not  complete,  fully  used  or  updated  and  no  perceived 
benefits  of  improving  the  software  architectures  developed. 

Incentives  specifically  for  software  were  not  used  by  most  of  the  subjects’ 
organizations.  Those  that  identified  incentives  for  contracts  did  not  specify  if  they  were 
for  development  of  the  program’s  software.  Interviewees  suggested  it  would  be 
challenging  to  have  incentives  for  quality,  reuse  and  application  of  commercial  best 
practices  due  the  difficulty  in  quantifying  contractor  efforts  in  these  areas. 

Interviewees  were  asked  if  their  program  had  a  standard  cost  estimation  model. 
This  question  resulted  with  many  different  cost  models  identified.  Some  responded  their 
program  used  two  different  models;  others  only  identified  one.  Some  of  the  common 
models  that  were  identified  were  the  Constructive  Cost  Model  (Cocomo),  Price  S, 
Software  Evaluation  and  Estimation  of  Resources-Software  Estimation  Model 
(SEER/SEM),  and  the  Automated  Cost  Estimating  Integrated  Tool  (ACEIT.)  Most 
agreed  that  there  should  be  multiple  models  to  compare  results  for  accuracy. 

There  was  no  consensus  on  tracking  software  cost  throughout  the  lifecycle  of  the 
program.  It  was  suggested  that  costs  were  captured  but  not  specific  to  software  and  were 
at  a  higher  level.  Other  suggested  that  software  cost  were  tracked  but  throughout  the 
entire  lifecycle.  The  portion  of  the  lifecycle  where  costs  were  tracked  was  program 
specific.  Others  suggested  no  software  costs  were  captured  throughout  the  lifecycle. 

All  interviewees  stated  their  program  had  a  risk  management  plan.  However, 
various  answers  were  given  on  whether  there  was  policy  how  to  create  them. 
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Interviewees  suggested  there  were  some  specific  polices  from  the  ASC  engineering  home 
office,  others  suggested  there  was  loose  policy  describing  the  creation  of  risk 
management  plans.  The  level  of  review  for  risk  management  plans  varied  from  bi¬ 
weekly  to  monthly.  Depending  on  the  level  in  the  program  reviews,  it  could  occur  more 
or  less  frequently. 

Interviewees  indicated  that  they  relied  on  their  contractor  or  the  engineering 
community  to  help  determine  program  deliverable  requirements.  Consensus  indicated 
that  engineers  should  help  the  program  managers  choose  deliverables  either  using  past 
experience  or  home  office  recommendations.  The  goals  of  each  deliverable  are  to  gain 
insight  into  contractor  efforts  and  to  deliver  useful  end-items. 

Only  one  individual  stated  his  program  had  a  Computer  Resource  Working  Group 
(CRWG).  Three  others  indicated  that  they  had  something  similar  to  what  this  group  was 
originally  intended  for.  However,  the  remaining  subjects  stated  this  group  was  largely 
out-dated  and  was  unnecessary  to  have. 

Interviewees  were  asked  if  their  program  had  an  independent  expert  review. 
Nearly  all  replied  that  their  program  had  an  independent  review  of  one  type  or  another. 
This  review  may  have  been  internal  or  through  external  sources.  Those  that  did  not  have 
an  independent  review  on  their  current  program  agreed  they  still  existed  and  may  have 
occurred  on  previous  programs.  It  was  also  suggested  that  the  ASC  engineering  home 
office  has  “Tiger  Teams”  to  provide  these  types  of  software  reviews. 
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Metrics 


Interviewees  we  asked  a  variety  of  questions  on  the  software  metrics  provided  by 
the  contractor.  This  resulted  in  many  different  metrics  provided  to  the  researcher, 
although  they  varied  program  to  program.  Their  use  centered  on  oversight  of  the 
contractor.  These  metrics  were  used  on  a  monthly  basis  at  lower  levels  of  the  program 
and  quarterly  at  higher  levels.  Interviewees  indicated  that  there  should  not  be  a  standard 
set  of  metrics,  but  rather  a  list  of  recommended  metrics.  It  would  then  be  up  to  the 
program  office  to  choose  from  the  list  which  metrics  to  receive.  This  choice  would  allow 
for  the  correct  oversight  for  each  individual  program. 

Though  many  different  answers  were  given  on  how  to  measure  success  of 
programs,  two  themes  arose.  The  first  was  that  programs  are  measured  via  successful 
achievement  of  cost,  schedule  and  performance  measures.  Program  failure  is  the  measure 
of  deviation  between  actual  and  planned  costs,  schedule,  and  performance.  The  other 
theme  that  surfaced  was  an  assessment  of  whether  the  system  works  as  defined  by  the 
user  representative.  Finally,  it  was  agreed  by  all  interviewees  that  contractors  should 
have  earned  value  management  (EVM)  systems  for  all  but  the  smallest  efforts  and  those 
with  fixed  price  contracts. 

Personnel 

Regarding  the  number  and  expertise  of  program  software  personnel,  half  of  the 
participants  indicated  that  their  programs  had  enough  engineers  and  managers  with 
software  experience  to  accomplish  their  programs.  The  other  half  stated  their  programs 
lacked  the  experienced  people  to  complete  their  program.  In  order  to  gain  the  required 
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knowledge  for  their  program,  participants’  organizations  relied  fairly  evenly  on  either 
center  engineering  staff  or  contract  experts. 

It  was  suggested  that  the  DoD  reduce  in-house  software  development  and  limit  it 
to  critical  functions  such  as  special  security-sensitive  work  (DSB:  1989).  This  idea  was 
presented  during  the  interviews  and  it  was  widely  agreed  upon  by  those  interviewed  that 
this  should  happen  and  in  most  cases  have  already  occurred.  It  was  stated  that  the  DoD 
has  gotten  out  of  the  development  business  and  due  to  cost  it  is  better  for  the  contractors 
to  do  the  development  and  maintenance  of  the  software. 

There  was  no  definite  common  answer  to  whether  or  not  contractor  maturity, 
design/code  reviews  and  V&V  were  accomplished  using  program  office  “in-house” 
personnel.  This  came  down  to  each  program  being  different.  Each  type  of  review  was 
done  in-house,  but  no  program  conducted  all  these  review  functions  without  outside 
assistance.  Most  thought  it  would  be  beneficial  to  do  these  types  of  measurements  in- 
house.  However,  responses  were  split  on  whether  the  program  required  additional 
personnel  to  accomplish  these  tasks.  Some  programs  planned  for  them  while  others 
relied  on  additional  support. 

Interviewees  also  indicated  that  programs  benefit  from  having  software  personnel 
to  stay  with  the  program  longer.  A  few  numbers  were  given  in  the  answers  that  were 
received;  however  a  common  answer  was  “long  enough.”  This  would  allow  for 
continuity  and  the  ability  to  retain  corporate  knowledge.  This  also  depends  on  the  person 
as  it  might  be  beneficial  to  the  program  for  those  dragging  down  the  program  to  leave 
earlier  than  planned.  It  was  also  mentioned  that  is  beneficial  for  individuals  to  work 
multiple  programs  in  order  to  gain  a  breadth  of  knowledge. 
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It  was  recommended  that  personnel  should  rotate  between  the  program  office  and 
the  developing  contractor  (DSB:  1994).  Interviewees  agreed  to  the  benefit  in  this 
recommendation,  but  that  it  would  be  administratively  difficult  to  accomplish.  This 
process  would  have  many  costs  associated  including  overhead  and  lost  productivity. 
Others  stated  that  there  might  be  a  benefit,  but  feared  there  was  a  risk  that  those  who 
rotated  would  be  looked  at  as  an  outcast  and  therefore  would  not  provide  much  benefit. 
The  advantage  of  doing  this  was  a  better  understanding  of  different  points  of  views  and 
gaining  insight  into  their  processes.  Insight  into  each  side’s  processes  was  also  seen  as  a 
negative,  because  those  processes  should  not  always  be  shared. 

Early  user  involvement  was  viewed  as  exceptionally  important.  The  user 
representative  should  be  brought  in  early  to  develop  requirements  and  to  ensure  it  is 
given  the  correct  priority.  However,  the  actual  level  of  user  involvement  throughout  the 
program  varied  drastically  on  each  program.  Most  that  the  user  representative  was  not  as 
involved  as  should  be  and  most  would  definitely  like  to  see  involvement  escalate  beyond 
participating  in  major  program  reviews. 

Test  and  Evaluation 

No  clear  consensus  was  discovered  when  participants  about  what  constitutes 
thorough  DT&E.  Many  suggested  the  testing  of  user  requirements.  Along  with  testing 
of  requirements,  it  was  suggested  the  DT&E  can  be  considered  thorough  through  trial  and 
error  and  the  use  of  experience  teams  that  come  to  a  consensus  on  the  level  of  maturity  of 
the  system. 
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Two  themes  arose  when  participants  were  asked  if  software  should  be  directly 
fielded  from  test  beds.  Interviewees  thought  it  possible  if  the  user  representative  agreed 
and  some  evaluation  of  the  test  beds  operational  representativeness  could  be  made. 

Others  believed  that  it  really  depended  on  the  application  of  the  software.  For  some 
systems  especially  business  systems  it  didn’t  matter  when  they  were  released.  However, 
when  it  comes  to  safety  critical  systems  in  may  not  be  the  best  to  release  these  systems 
prior  to  extensive  testing. 

The  question  was  raised:  Were  future  maintainers  being  brought  in  to  do  V&V 
during  software  development?  Respondents  divided  on  whether  or  not  maintainers  were 
brought  in  early  in  software  development.  One  reply  was  that  on  the  current  program  the 
maintainers  were  brought  it  to  do  the  V&V,  yet  on  their  previous  program  this  did  not 
occur.  The  other  person  had  the  opposite  situation  occur. 

The  final  question  in  regards  to  test  and  evaluation  was:  Who  is  going  to  perform 
the  Operational  Test  &  Evaluation  on  your  program  and  were  facilities  provided  to  them? 
In  a  majority  of  the  cases  the  Air  Force  Operational  Test  and  Evaluation  Center 
(AFOTEC)  would  conduct  the  OT&E.  If  AFOTEC  was  not  conducting  the  OT&E,  it 
was  performed  by  a  MAJCOM  testing  organization.  In  all  cases  facilities  were  provided 
or  already  established. 
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Support 

The  questions  regarding  support  focused  on  who  was  going  to  maintain  the 
software  and  how  was  the  software  going  to  be  maintained.  It  was  agreed  that  software 
maintenance  issues  need  to  be  covered  early  in  the  life  cycle.  In  many  cases  a  plan  was 
developed  to  cover  software  maintenance  issues.  These  plans  were  developed  by  the 
groups  responsible  for  maintaining  the  systems  software.  For  most,  the  contractor  would 
maintain  the  software  after  development  was  complete.  In  order  to  release  new  software 
to  get  the  new  software  fielded,  a  variety  of  different  methods  were  suggested  by  the 
interviewees.  These  included  blocks,  suites  and  other  tailored  processes  developed  by  the 
contractor.  To  track  and  identify  problems  for  fixes  to  be  incorporated  into  these 
releases,  respondents  stated  that  there  were  many  different  fonnal  processes.  These  could 
be  controlled  either  by  the  government  or  the  contractor. 

Summary 

A  variety  of  answers  were  received  during  the  interviews  due  to  the  use  of  open- 
ended  interview  questions.  Throughout  the  analysis  of  the  conducted  interviews  many 
themes  and  patterns  arose  in  the  collected  data.  This  showed  that  in  many  cases  a 
majority  of  the  interviewees  had  similar  experiences  in  software  development  and 
acquisitions.  The  next  chapter  will  provide  conclusions  and  recommendations  of  the 
researcher  on  the  data  collected  and  described  in  chapter  4  of  this  study. 
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V.  Conclusions  &  Recommendations 


Overview 

The  purpose  of  this  research  was  to  confirm  the  application  of  previous 
recommendations  to  improve  software  acquisition  in  ASC  and  to  investigate  any 
perceived  benefits.  This  study  was  accomplished  through  interviews  of  practitioners  in 
the  software  development  field  and  through  a  review  of  literature  relating  to  this  study. 
This  chapter  presents  conclusions  from  this  research. 

Policy 

This  study  found  that  acquisition  policy  related  to  software  intensive  systems  has 
continued  to  grow  and  evolve.  This  continual  growth  and  evolution  has  made  it  difficult 
for  software  practitioners  to  compile  a  clear  set  of  policy  required  for  their  programs.  In 
order  to  better  serve  program  managers  it  is  recommended  that  the  DoD  assimilate 
regulations  and  produce  a  central  source  of  authoritative  policy.  This  will  serve  as  a 
“one-stop-shop”  for  program  mangers  of  software  intensive  systems. 

Research  and  Development 

The  commercial  market  is  driving  technology  today  and  the  DoD  must  stay  in 
close  contact  with  the  market  in  order  to  leverage  state-of-the-art  technology.  Due  to 
unique  and  ever  changing  requirements  the  DoD  cannot  always  depend  on  the 
commercial  market  to  meet  their  unique  needs.  Based  on  the  results  of  this  study,  the 
DoD  should  continue  to  create  research  programs  in  order  to  obtain  the  necessary 
knowledge  and  products  to  meet  the  specific  needs. 
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Best  Practice 


Recommendations  have  been  made  to  collect  and  disseminate  both  government 
and  industry  best  practices.  Based  on  the  interviews  conducted,  collection  and 
dissemination  of  best  practices  has  not  been  fully  implemented.  A  recommended  course 
of  action  is  to  implement  a  central  repository,  including  a  searchable  database,  for  the 
collection  of  best  practices.  All  individuals  working  in  the  development  and  acquisitions 
of  software-intensive  systems  should  be  aware  of  this  repository.  The  Defense 
Acquisition  University  currently  has  a  website  dedicated  to  sharing  acquisition 
knowledge.  One  specific  community  concerned  with  software  acquisition  management  is 
located  at  https://acc.dau.mil/sam,  intends  to  share  concerns,  policies,  and  practices  to 
assist  others  in  software  development.  It  is  recommended  that  this  site  or  a  similar  one  be 
expanded  and  further  promoted  as  an  educational  tool  for  DoD  software  personnel. 

These  websites  should  be  advertised  and  emphasized  at  the  program  office  to  increase 
participation.  It  should  also  have  a  panel  of  software  experts  that  serve  as  moderators  to 
ensure  all  questions  and  suggestions  are  dealt  with  appropriately. 

Lifecycle 

Though  it  was  agreed  that  Evolutionary  Acquisition  should  be  used  for  software 
acquisition  and  development,  it  is  not  the  only  model  available  for  use.  Respondents 
were  not  clear  on  policy  or  guidance  describing  how  to  choose  or  tailor  lifecycle  process 
to  their  program.  It  is  a  recommendation  that  clear  guidance  and  possible 
recommendations  of  lifecycles  should  be  developed  to  allow  program  office  to 
adequately  choose  the  proper  lifecycle  to  fit  their  program. 
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Source  Selection 


Many  of  those  interviewed  discussed  government  contractors  obtaining  various 
maturity  levels.  The  most  common  answer  common  answer  received  was,  “CMMI  Level 
3.”  However,  it  was  also  not  understood  how  maturity  levels  should  be  considered 
during  source  selection.  It  is  recommended  there  be  clear  guidance  established  and  key 
parameters  developed  for  using  maturity  as  a  criterion  in  a  source  selection. 

COTS 

To  better  leverage  the  commercial  market  in  the  development  of  DoD  systems,  a 
stronger  emphasis  has  been  placed  on  the  use  of  COTS  software  products.  These 
products  pose  both  pros  and  cons  with  their  use  in  DoD  systems.  Recommendations 
involving  COTS  products  were  made  based  on  this  emphasis  and  suggested  that  COTS 
products  be  looked  at  to  meet  software  requirements. 

COTS  products  are  not  without  risks;  program  managers  should  not  assume  that 
requirements  can  be  met  with  COTS  products.  Rather,  COTS  or  modified  COTS  should 
only  be  considered  if  they  meet  the  necessary  requirements  and  if  justified  through  a  life- 
cycle  analysis.  Difficulty  finding  COTS  products  to  meet  program  requirements  was  also 
discovered.  Many  different  ways  to  discover  COTS  products  were  discussed. 
Recommended  actions  include  drafting  a  policy  requiring  the  use  of  life-cycle  analysis  to 
evaluate  COTS  products  and  creating  a  database  of  potential  COTS  products.  This 
database  should  include  recommendations  of  other  programs  that  may  have  considered 
the  use  of  a  particular  product. 
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Program  Management 

It  was  determined  that  respondents  do  not  understand  software  architectures  or  the 
perceived  benefit  of  them.  Most  were  not  aware  of  an  architecture  in  their  program. 
Those  that  were  aware  stated  that  it  was  not  at  the  software  level.  It  was  not  clear  if  those 
interviewed  even  felt  there  was  a  benefit  to  having  a  software  architecture.  This  study 
recommends  that  software  practitioners  be  educated  on  the  development  and  use  of 
software  architectures. 

Metrics 

It  was  common  for  programs  to  use  a  variety  of  metrics  to  track  progress  in  their 
programs,  from  the  program  level  to  the  software  level.  To  most  this  was  beneficial  and 
practical.  However,  cost  data  was  not  shown  to  be  tracked  at  the  software  level.  Software 
costs  were  even  shown  not  to  be  covered  for  the  entire  lifecycle.  This  level  of  tracking 
could  provide  benefit  to  programs.  The  benefit  received  is  unknown.  Therefore,  it  is  a 
recommendation  to  conduct  a  study  and  determine  the  level  of  cost  data  required  to 
benefit  software  acquisition  programs. 

Personnel 

It  could  not  be  concluded  if  there  is  a  lack  of  personnel  with  software  expertise. 
The  opinions  of  those  interviewed  varied;  some  interviewees  thought  they  had  adequate 
personnel  for  their  programs,  while  others  stated  the  opposite.  There  is  great  disparity  in 
what  is  adequate  staffing  for  individual  software  programs.  It  is  recommended  to  further 
study  the  number  of  personnel  required  for  each  program.  This  study  should  include  not 
only  the  number  of  personnel,  but  also  the  type  of  expertise. 
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Support 

In  questions  concerning  Test  &  Evaluation  and  support,  it  was  discovered  that 
software  maintenance  issues  are  important  and  need  to  be  addressed  early  in  the  program. 
The  majority  of  respondents  indicated  that  their  programs  have  elected  to  have 
contractors  maintain  their  software.  It  was  a  common  answer  that  the  DoD  does  not  have 
organic  support.  However,  beside  opinions,  it  was  not  clear  if  there  was  a  precise  reason 
that  programs  choose  contractor  support  over  organic  support.  Decision  criteria  should 
be  developed  and  benefits  should  be  weighed  in  the  detennination  of  contractor  or 
organic  support  in  software  development. 

Application  of  Recommendations 

The  analysis  described  in  chapter  four  confirms  many  of  recommendations 
investigated  in  this  study  have  been  applied  in  programs  at  ASC.  The  recommendations 
may  have  not  been  applied  in  their  entirety  or  as  originally  intended  and  in  some  case 
they  have  been  updated  due  to  the  age  of  the  recommendation.  Updating  the 
recommendations  was  required  due  to  change  in  policy,  technology  and  the  business 
environment  over  the  past  years  since  the  recommendations  were  originally  made.  It 
cannot  be  confirmed  that  the  use  of  these  recommendations  benefit  all  programs. 
Perceived  benefit  varied  from  program  to  program;  some  saw  no  benefit  to  the  use  of  the 
recommendations. 

It  is  therefore  an  overall  recommendation  of  this  study  to  consider  these 
recommendations  not  as  policy,  but  as  best  practices.  The  recommendations  should  not 
be  forced  on  an  organization,  but  made  available  as  options  to  improve  their  development 
and  acquisition  of  software.  Specific  recommendations  do  not  provide  an  overall  benefit 
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to  every  program.  This  further  justifies  allowing  each  program  to  choose  what  best 
benefits  the  program. 

Limitations  of  Study 

This  study  was  conducted  mostly  at  ASC  with  limited  input  from  the  554th  ELSG. 
Though  no  significant  difference  was  recognized  between  the  organizations  only  a  small 
number  of  individuals  were  interviewed.  The  lack  of  differences  between  these  two 
organizations  may  begin  to  indicate  that  the  results  may  be  consistent  across  the  Air 
Force.  Interviewing  a  larger  number  of  practitioners  from  several  organizations  may 
provide  different  results  or  more  consistent  results  across  the  Air  Force. 

To  reduce  the  scope  of  this  study,  only  recommendations  by  the  DSB,  NRC,  and 
GAO  were  considered.  These  organizations  are  commonly  tasked  by  Congress  to 
conduct  studies  of  DoD  programs.  There  are  many  different  organizations  producing 
recommendations  to  improve  software  development,  too  numerous  to  be  considered  in 
the  scope  of  this  research. 

The  large  number  of  recommendations  considered  resulted  in  a  larger  number  of 
interview  questions.  This  translated  into  a  longer  interview.  It  was  an  observation  of  the 
researcher  that  as  the  interview  time  increased  participants’  responses  became  shorter. 
Therefore,  interviewees  became  less  involved  in  the  interview.  As  a  result,  it  was  more 
difficult  to  draw  conclusive  results  from  the  later  portion  of  the  responses. 

Future  Research 

To  further  investigate  the  findings  found  in  this  study  it  is  recommended  that  a 
follow-on  quantitative  study  be  undertaken.  Using  the  data  in  chapter  four,  survey 
questions  can  be  developed  and  delivered  to  a  larger  sample  of  DoD  software  personnel. 
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This  sample  could  include  the  entire  DoD  or  specific  services.  If  a  quantitative  study  is 
developed,  future  research  can  add  generalizations  to  the  entire  DoD  or  specific  services 
on  the  application  and  perceived  benefits  to  recommendations  considered  in  this  study. 

It  is  also  recommended  that  future  research  evaluate  recommendations  from  other 
government  agencies,  professional  organizations,  and  industry.  This  will  allow  for 
different  points  of  view  on  ways  to  improve  software  development.  However,  with  the 
large  number  of  recommendations  future  research  should  be  reduced  in  scope  to  focus  on 
specific  areas  of  software  acquisition  and  development. 

Summary 

The  purpose  of  this  research  was  to  confirm  the  application  of  previous 
recommendations  to  improve  software  development  in  ASC  and  to  investigate  any 
perceived  benefits.  This  research  found  that  previous  recommendations  were  applied  in 
numerous  programs  at  ASC.  It  was  also  concluded  that  recommendations  were  not 
universally  applied  to  all  programs  since  there  was  not  a  perceived  benefit  in  all 
programs.  In  conclusion,  this  research  found  that  some  of  the  same  problems  facing 
software  development  in  the  1980s  -1990s  are  still  relevant  today.  Though  the  issues  are 
still  relevant,  the  recommended  solutions  of  the  past  may  not  be  the  universal  solution  to 
correct  the  problems  of  today.  It  is  therefore  recommended  that  the  issues  on  software 
development  be  continually  evaluated  and  that  best  practices  be  applied  to  improve  the 
software  acquisition  and  development  environment  within  ASC. 
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Appendix  A.  Interview  Questionnaire 


Software  Acquisition  and  Development 


Policy 

1 .  Is  there  a  clear  set  of  Acquisition  Policy  for  software  development? 

a)  Is  there  enough  up-to-date  guidance  for  accomplishing  your  program? 

2.  Have  you  seen  impacts  from  Section  804? 

R&D 

3.  It  has  been  suggested  that  the  commercial  market,  not  the  DoD,  is  clearly  driving 
today’s  information  technology.  DoD,  however,  must  stay  abreast  of  the  most 
current  technology,  and  are  there  areas  that  are  imperative  to  the  success  of  DoD 
which  are  not  being  cover  by  the  commercial  market? 

4.  In  addition  to  looking  at  COTS  products  should  we  continue  to  be  looking  for 
opportunities  to  buy,  in  the  civilian  market,  tools,  methods,  environments,  and 
application  software? 

Best  Practices 

5.  In  your  experience  has  the  DoD  been  effective  at  collecting  and  disseminating  best 
practices  of  both  the  government  and  industry? 

6.  Is  there  a  process  to  evaluate  the  usage  of  best  practices? 

Lifecycle 

7.  Is  it  appropriate  to  use  Evolutionary  Acquisition,  including  simulation  and 
prototyping,  for  software  development? 

a)  Should  it  be  the  primary  model? 
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8.  Should  the  Software  Acquisition  Process  be  standardized  or  tailored  for  each 
system? 

a)  Based  on  classifications  such  as:  Life  cycle  Model,  Requirement  Stability, 
Reuse  potential,  Contract  and  Support  Strategy,  and  %  of  new  development, 
COTS,  Modified  COTS,  or  Custom? 

b)  Should  programs  be  tailored  based  on  amount  of  user  involvement? 

c)  Is  there  policy/guidance  on  how  this  should  be  done? 

Source  Selection 

9.  Is  there  a  policy  requiring  government  and/or  contractor  software-intensive 
acquisition  projects  to  reach  a  particular  capability  maturity  level  or  equivalent? 

a)  Is  this  policy  helping  improve  the  development  of  these  programs? 

10.  Is  evaluating  competitors  on  their  technical  approach  rather  than  cost,  feasible? 

1 1 .  Is  requiring  the  Government,  prior  to  RFP,  to  perform  an  Independent  Market 
Analysis  of  Off-the-Shelf  and  contractor  products  to  assure  a  “Best  Value”  solution 
beneficial  to  the  acquisition  of  software? 

12.  When  considering  offers  in  source  selection  process  should  the  offerors  be 
encouraged  to  demonstrate  as  much  pre-existing  functionality  as  possible? 

a)  How  was  experience  with  COTS  usage  considered  in  source  selection? 


COTS 

13.  Do  you  have  a  system  to  help  identify  potential  COTS  products  to  meet  systems 
requirements? 

14.  Should  Program  Managers  assume  that  system  software  requirements  can  be  met 
with  off-the-shelf  subsystems  and  components  until  it  is  proved  that  they  are 
unique? 

15.  Should  the  modification  of  commercial  components  be  discouraged  and  allowed 
only  if  justified  by  a  thorough  analysis  of  life-cycle  costs  and  benefits? 
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Project  Management 


16.  Is  it  beneficial  to  have  program  managers  manage  price,  schedule,  and  functionality 
but  only  constrain  2  of  the  3? 

a)  Is  this  a  realistic  approach? 

17.  How  do  you  know  your  software  requirements  are  feasible? 

a)  How  do  you  know  you  and  your  contractor  have  the  same  understanding  of  the 
requirements? 

b)  Was  a  trade-off  analyses  performed,  supported  by  systems  engineering  analysis, 
considering  performance,  cost,  and  schedule  impacts  of  major  changes  to 
software  requirements? 

18.  Who  should  have  the  authority  to  defer  requirements? 

19.  What  is  the  role  of  software  architectures  in  your  program? 
a)  Has  it  improved  your  software  development? 

20.  Do  you  use  incentives  on  contract  for  the  contractor  to  build  better  software? 

a)  Should  there  be  incentives  for  quality,  reuses,  and  application  of  commercial 
best  practices? 

2 1 .  Is  there  a  standard  cost  estimation  model  used  for  your  program? 

22.  Does  your  program  track  actual  software  cost  throughout  the  entire  lifecycle? 

23.  Do  you  have  an  established  Risk  Management  Plan? 

a)  Is  there  policy  on  how  a  RMP  should  be  created? 

b)  When  are  program  risks  reviewed? 

24.  How  do/did  you  decide  what  software  engineering  deliverables  to  require? 

25.  Does  your  program  have  a  CRWG  or  similar  IPT? 

a)  How  is  the  performance  of  that  group  evaluated? 

26.  Have  you  had  an  Independent  Expert  Review  (IER)? 
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Metrics 


27.  What  metrics  are  provided  to  you  by  the  contractor? 

a)  How  do  you  use  them? 

b)  How  often  are  they  used? 

c)  Would  it  be  better  to  get  them  more  often? 

d)  Would  it  benefit  you  to  get  metrics  related  to  cost,  schedule,  size,  requirements, 
tests,  defects,  and  quality  to  program  offices  on  a  monthly  basis  and  before 
program  milestones? 

28.  Would  the  use  of  standardized  Software  maturity  metrics  be  beneficial  in  the 
development  of  software? 

29.  How  do  you  measure  success  in  your  program? 

30.  Should  the  DoD  ensure  that  contractors  have  an  earned  value  management  system 
that  reports  cost  and  schedule  information  at  a  level  of  work  that  provides 
infonnation  specific  to  software  development? 

Personnel 

3 1 .  Do  you  have  the  appropriate  number  of  software  personnel  with  the  right  skills  for 
your  program? 

32.  Do  you  have  enough  in-house  (within  SPO)  software  expertise  or  do  you  rely  on 
center  engineering  staff  or  contract  out  for  software  expertise? 

33.  Should  the  DoD  reduce  in-house  software  construction,  extension,  and 
maintenance,  limiting  such  to  critical  functions  at  operational  bases,  adaptation  of 
existing  software  to  local  needs,  and  special  security-sensitive  work? 

34.  Which  of  the  following  tasks  is  done  in-house:  Contractor  maturity  measurement, 
design/code  reviews,  and  V&V? 

a)  Was  it  beneficial  to  do  them  in-house? 

b)  Did  it  require  additional  resources? 

35.  Would  it  be  beneficial  for  program  office  personnel  to  stay  with  the  program 
longer? 

36.  Would  the  rotation  of  government  and  contractor  personnel  between  the  PM  and  the 
developing  organization  be  beneficial  to  software  development? 
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37.  How  important  is  early  user  involvement  in  software  development  programs  and 
what  is  the  nature  of  the  relationship  in  your  program? 

a)  How  involved  are  they? 


Test  and  Evaluation 


38.  How  do  you  know  what  constitutes  thorough  DT&E? 

39.  Should  software  be  directly  fielded  from  test  beds  if  given  user  consent? 

40.  Where  future  maintainers  of  your  software  product  brought  in  to  do  V&V  during 
software  development? 

41.  Who  is/will  be  perfonning  the  Operational  Test  and  Evaluation? 

a)  Have  facilities  been  provided  for  the  completion  of  this  testing? 

Support 

42.  Do  software  maintenance  issues  need  to  be  covered  earlier  in  the  lifecycle? 

43.  Who  is  going  to  maintain  your  software  system? 

a)  How  do  you  evaluate  the  efficiency/benefits  of  in-house  software  support  versus 
contractor  software  support? 

44.  Was  a  plan  developed  for  software  maintenance? 

45.  Do  you  have  a  designed  process  for  release  of  software  that  is  ready  to  be  fielded, 
block  increments,  or  improvements? 

a)  Has  it  helped  reduce  cycle  time  in  development  and  release  of  the  software? 

46.  Do  you  have  a  formal  process  to  identify,  track,  and  assign  problems  in  your 
software  development? 
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—  Backup  Material  — 


National  Defense  Authorization  Act  of  2003  -SECT.804 

A.  Establishment  of  Program 

(1)  The  Secretary  of  each  military  department  shall  establish  a  program  to 
improve  the  software  acquisition  processes  of  that  military  department. 

(2)  The  head  of  each  Defense  Agency  that  manages  a  major  defense  acquisition 
program  with  a  substantial  software  component  shall  establish  a  program  to 
improve  the  software  acquisition  processes  of  that  Defense  Agency. 

(3)  The  programs  required  by  this  subsection  shall  be  established  not  later  than 
120  days  after  the  date  of  the  enactment  of  this  Act. 

B.  Program  Requirements. — a  program  to  improve  software  acquisition  processes  under 
this  section  shall,  at  a  minimum,  include  the  following: 

(1)  A  documented  process  for  software  acquisition  planning,  requirements 
development  and  management,  project  management  and  oversight,  and 
risk  management. 

(2)  Efforts  to  develop  appropriate  metrics  for  performance  measurement  and 
continual  process  improvement. 

(3)  A  process  to  ensure  that  key  program  personnel  have  an  appropriate  level 
of  experience  or  training  in  software  acquisition. 

(4)  A  process  to  ensure  that  each  military  department  and  Defense  Agency 
implements  and  adheres  to  established  processes  and  requirements  relating 
to  the  acquisition  of  software. 
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Appendix  B.  Recommendations 


Recommendation  Source  Reason  for 


Exclusion 


1 

All  the  methodological  efforts,  especially  STARS, 
should  look  to  see  how  commercially  available 
software  tools  can  be  selected  and  standardized  for 
DoD  needs 

DSB 

1987 

2 

DoD  should  examine  and  revise  regulations  to 
approach  modern  commercial  practices  insofar  as 
practicable  and  appropriate. 

DSB 

1987 

3 

Direct  STARS  to  choose  several  real  programs  early 
in  development  and  augment  their  funding  to  ensure 
the  use  of  existing  practices  and  tools. 

DSB 

1987 

4 

Use  evolutionary  acquisition,  including  simulation  and 
prototyping,  as  discussed  else  ware  in  this  report,  to 
reduce  risk. 

DSB 

1987 

5 

The  Undersecretary  of  Defense  (Acquisition)  should 
update  DoD  Directive  5000.29,  "Management  of 
Computer  Resources  in  Major  Defense  Systems",  so 
that  it  mandates  the  iterative  setting  of  specifications, 
and  rapid  prototyping  of  specified  systems  and, 
incremental  development. 

DSB 

1987 

6 

DoD  STD  2167  should  be  further  revised  to  remove 
any  remaining  dependence  upon  the  assumption  of 
the  "waterfall"  model  and  institutionalize  rapid 
prototyping  and  incremental  development. 

DSB 

1987 

7 

Each  service  should  provide  its  software  Product 
Development  Division  with  the  ability  to  do  rapid 
prototyping  in  conjunction  with  users. 

DSB 

1987 

8 

The  Undersecretary  of  Defense  (Acquisition)  should 
adopt  a  four  category  classification  as  a  bias  for 
acquisition  policy:  Standard,  Extended,  Embedded, 
and  Advanced. 

DSB 

1987 

9 

The  Undersecretary  of  Defense  (Acquisition)  should 
develop  acquisition  policy,  procedures,  and  guidance 
for  each  category  (Follow  on  to  above 
recommendation) 

DSB 

1987 

10 

The  Undersecretary  of  Defense  (Acquisition)  and  the 
Assistant  Secretary  of  Defense  (Comptroller)  should 
direct  Program  Managers  to  assume  that  systems 
software  requirements  can  be  met  with  off-the-shelf 
subsystems  and  components  until  it  is  proven  that 
they  are  unique. 

DSB 

1987 
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11 

The  Undersecretary  of  Defense  (Acquisitions)  and 
the  Assistant  Secretary  of  Defense  (Comptroller) 
should  by  directive  spell  out  the  role  of  Using 
Commands  in  the  evolutionary  and  incremental 
development  of  software  systems. 

DSB 

1987 

12 

DoD  should  devise  increased  profit  incentives  on 
software  quality. 

DSB 

1987 

13 

The  Undersecretary  of  Defense  (Acquisition)  and  the 
Assistant  Secretary  of  Defense  (Comptroller)  should 
direct  Program  Managers  to  identify  in  their  programs 
those  subsystems,  components  and  perhaps  even 
modules,  that  may  be  expected  to  be  acquired  rather 
than  built;  and  to  reward  such  acquisition  in  the 

RFP's. 

DSB 

1987 

14 

The  Undersecretary  of  Defense  (Acquisition)  should 
develop  economic  incentives,  to  be  incorporated  into 
standard  contracts,  to  allow  contractors  to  profit  from 
offering  modules  for  reuse,  even  though  built  with 

DoD  funds. 

DSB 

1987 

15 

The  Undersecretary  of  Defense  (Acquisition)  should 
develop  economic  incentives,  to  be  incorporated  into 
all  cost-plus  standard  contracts,  to  encourage 
contractors  to  buy  modules  and  use  them  rather  than 
building  new  ones. 

DSB 

1987 

16 

DoD  should  devise  increased  productivity  incentives 
for  custom-built  software  contracts,  and  make  such 
incentives  contracts  the  standard  practice. 

DSB 

1987 

17 

Directive  5000.29  and  STD  2167  should  be  revised  or 
superseded  by  policy  mandate  risk  management 
techniques  in  software  acquisition,  as  recommended 
in  1983  USAF/SAB  Study. 

DSB 

1987 

18 

DoD  should  develop  metrics  and  measuring 
techniques  for  software  quality  and  completeness, 
incorporate  these  routinely  in  contracts. 

DSB 

1987 

19 

Focus  a  critical  mass  of  software  research  effort  on 
software  needs  that  are  unique  to  SDI  objectives. 

DSB 

1987 

20 

Task  the  new  STARS  director  to  define  a  new  set  of 
program  goals  together  with  an  implementation  plan; 
emphasis  should  be  on  visible,  early  milestones  that 
have  demonstratable  results 

DSB 

1987 

21 

DoD  should  develop  metrics  to  measure 
implementation  progress. 

DSB 

1987 

22 

Each  service  should  provide  its  software  Using 
Commands  with  facilities  to  do  comprehensive 
operational  testing  and  life-cycle  evaluation  of 
extensions  and  changes. 

DSB 

1987 
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23 

Task  the  STARS  Office,  the  Ada  JPO,  and  SEI,  the 

SDI  software  methodology  program  element,  and 
DARPA  Strategic  Computing  Program  to  produce  a 
one-time  joint  plan  to  demonstrate  a  coordinated 
Software  Technology  program. 

DSB 

1987 

Due  to  decreasing 
support  by  industry 
and  academia  the 
mandate  for  the 
required  use  of  Ada 
removed  and  the  Ada 
JPO  was  dissolved 

24 

Commit  DoD  management  to  serious  and  determined 
push  to  Ada 

DSB 

1987 

Due  to  decreasing 
support  by  industry 
and  academia  the 
mandate  for  the 
required  use  of  Ada 
removed  and  the  Ada 
JPO  was  dissolved 

25 

Move  the  Ada  JPO  into  the  same  organization  as 
STARS  and  the  SEI. 

DSB 

1987 

Due  to  decreasing 
support  by  industry 
and  academia  the 
mandate  for  the 
required  use  of  Ada 
removed  and  the  Ada 
JPO  was  dissolved 

26 

Keep  the  AJPO  as  the  technical  staff  support  agent 
for  the  DoD's  executive  agent. 

DSB 

1987 

Due  to  decreasing 
support  by  industry 
and  academia  the 
mandate  for  the 
required  use  of  Ada 
removed  and  the  Ada 
JPO  was  dissolved 

27 

DoD  policy  should  continue  to  forbid  subsetting  of 

Ada  language. 

DSB 

1987 

Due  to  decreasing 
support  by  industry 
and  academia  the 
mandate  for  the 
required  use  of  Ada 
removed  and  the  Ada 
JPO  was  dissolved 

28 

The  DoD  should  increase  investments  in  Ada 
practices  education  and  training,  for  both  technical 
and  management  people. 

DSB 

1987 

Due  to  decreasing 
support  by  industry 
and  academia  the 
mandate  for  the 
required  use  of  Ada 
removed  and  the  Ada 
JPO  was  dissolved 

29 

Allow  fourth-generation  languages  to  be  used  where 
the  full  life-cycle  cost-effectiveness  of  using  the 
language  measures  more  than  tenfold  over  the  using 
a  general-purpose  language. 

DSB 

1987 

Due  to  decreasing 
support  by  industry 
and  academia  the 
mandate  for  the 
required  use  of  Ada 
removed  and  the  Ada 
JPO  was  dissolved 

47 


30 

The  Software  Engineering  Institute  should  establish  a 
prototype  module  market,  focused  on  Ada  modules 
and  tools  for  Ada,  with  the  objective  of  spinning  it  off 
when  commercially  viable 

DSB 

1987 

Due  to  decreasing 
support  by  industry 
and  academia  the 
mandate  for  the 
required  use  of  Ada 
removed  and  the  Ada 
JPO  was  dissolved 

31 

The  Software  Engineering  Institute,  in  consultation 
with  the  Ada  JPO,  should  establish  standards  of 
Description  for  Ada  modules  to  be  offered  through  the 
Software  Module  Market. 

DSB 

1987 

Due  to  decreasing 
support  by  industry 
and  academia  the 
mandate  for  the 
required  use  of  Ada 
removed  and  the  Ada 
JPO  was  dissolved 

32 

DoD  should  follow  the  concepts  of  the  proposed  FAR 
27.4  for  data  rights  for  military  software,  rather  than 
those  of  the  proposed  DoD  Supplement  27.4  ,  or  it 
should  adopt  a  new  "Rights  in  Software"  see 
appendix  A6 

DSB 

1987 

FAR  27.4  currently 
used  includes 
regulations  on  data 
rights 

33 

Move  STARS  and  Rebuild  it. 

DSB 

1987 

STARS  is  no  longer  a 
program 

34 

SS-311  Establish  clear  Acquisition  Policy  for 

Software 

DSB 

1989 

35 

SS-316  Enhance  Interaction  between  Activities 

DSB 

1989 

36 

SS-315  Develop  a  Computer  Resource  Data  Base 

DSB 

1989 

37 

SS-114  Evaluate  Software  Life  Cycle  Models 

DSB 

1989 

38 

SS-133  Tailor  Software  Acquisition  Process  to 

Systems 

DSB 

1989 

39 

SS-134  Develop  a  Consistent  Contracting  Approach 

DSB 

1989 

40 

SS-1 12  Develop  an  Approach  to  Software  Reuse 

DSB 

1989 

41 

SS-123  Establish  mechanism  for  Reverse 

Engineering 

DSB 

1989 

42 

SS-41 1  Enforce  Standard  Software  Cost  Model 

DSB 

1989 

43 

SS-413  Identify  and  Capture  Actual  Software  Costs 

DSB 

1989 

44 

SS-326  Provide  Software  Maturity  Management 

DSB 

1989 

45 

SS-1 13  Develop  and  Evaluate  Software  Metrics 

DSB 

1989 

46 

SS-1 1 1  Implement  an  Effective  Software  R&D 

Strategy 

DSB 

1989 

47 

SS-1 31  Develop  a  Strategy  for  Technology  Insertion 

DSB 

1989 

48 

SS-431  Develop  Software  Engineering  Career 

Program 

DSB 

1989 

49 

SS-223  Organize  to  Grow  Software  Engineers 

DSB 

1989 

48 


50 

SS-232  Develop  Operational  Software  Literacy 
Program 

DSB 

1989 

51 

SS-432  Improve  Incentives  For  Military  Software 
Experts 

DSB 

1989 

52 

SS-433  Establish  Career  Subprogram  Management 

DSB 

1989 

53 

SS-434  Provide  Job  Challenge  for  Software 

Engineers 

DSB 

1989 

54 

SS-221  Provide  One-Stop  Support  for  Program 
Managers 

DSB 

1989 

55 

SS-423  Conduct  Contracting  Out  Study 

DSB 

1989 

56 

SS-421  Provide  Efficient  front  End  Loading 

DSB 

1989 

57 

SS-132  Conduct  Integrated  Software  Planning 

DSB 

1989 

58 

SS-424  Measure  Efficiency  of  Current  LCSE  Centers 

DSB 

1989 

59 

SS-31 3  Provide  for  Management  of  Software  Change 

DSB 

1989 

60 

SS-422  Consider  Alternative  Support  Options 

DSB 

1989 

61 

SS-324  Address  Software  as  part  of  a  Materiel 

Release 

DSB 

1989 

62 

SS-325  Develop  Responsive  Distribution  Processes 

DSB 

1989 

63 

SS-31 4  Establish  Internal  Controls  and  Feedback 

DSB 

1989 

64 

SS-321  Integrate  Software  Ouality  into  Process 

DSB 

1989 

65 

SS-312  Clarify  Funding  Policy  for  Software  Support 

DSB 

1989 

Funding  issue  is 
largely  OBE 

66 

SS-122  Manage  the  introduction  of  Ada  into  the  Army 

DSB 

1989 

Due  to  decreasing 
support  by  industry 
and  academia  the 
mandate  for  the 
required  use  of  Ada 
removed  and  the  Ada 
JPO  was  dissolved 

67 

SS-231  Develop  Pilot  Software  Awareness  Program 

DSB 

1989 

68 

SS-224  Eliminate  Confusion  in  Training  Device 

Support 

DSB 

1989 

Specific  to  AMC,  not 
an  issue  for  USAF 

69 

SS-225  Provide  Virtual  Collocation  with  TRADOC 
Centers 

DSB 

1989 

No  longer  relevant,  e- 
mail,  video 
conferencing, 
electronic  blackboards 
commonly  used 

70 

SS-21 1  Organize  Army  to  Manage  Acquisition 

Process 

DSB 

1989 

Specific  to  AMC 

71 

SS-212  Improve  PM/PEO  Computer  Resource 
Management 

DSB 

1989 

Specific  to  AMC 
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72 

SS-213  Establish  Clear  Organizational 

Responsibilities 

DSB 

1989 

Specific  to  AMC 

73 

SS-214  Strengthen  AMC'S  software  Management 

Role 

DSB 

1989 

Specific  to  AMC 

74 

SS-222  Build  an  Army  Software  Technology  Center 

DSB 

1989 

Specific  to  AMC 

75 

SS-233  Find  Army  Software  Advocates 

DSB 

1989 

Specific  to  AMC 

76 

SS-121  Establish  Controls  on  Software  Environments 

DSB 

1989 

DoD  no  longer  drives 

development 

technology 

77 

SS-322  Improve  Software  Configuration  Management 

DSB 

1989 

Best  Practices  and 

DoD  no  longer  driving 
technology 

78 

SS-323  Implement  Effective  Interoperability  Control 

DSB 

1989 

Basic  requirements 
issue  --  the  JCIDS 
process  drives  us 
toward  interoperability 
--  not  just  software 

79 

SS-412  Improve  Interface  into  PPBS  for  Software 

DSB 

1989 

Funding  issue  is 
largely  OBE 

80 

Emphasize  Technology  Transfer  (External  and 

Internal) 

-  Fund  technology  transfer  programs 

-  Initiate  demonstration  program  (e.g.,  ATDs)  to 
facilitate  software  technology  insertion  into  systems. 
Examples  of  candidate  criteria: 

-  Open  Standards,  Use  of  COTS  and  GOTS, 

Frequent  releases  to  include  numbers  of  users, 
multiple  platforms,  and  stratifies  commercial 
standards  and  interoperability  standards  across  DoD 
organization 

DSB 

1994 

81 

Increase  tech  base  funding  for  security  audit  tools  for 
systems  employing  COTS 

DSB 

1994 

82 

Establish  acquisition  focus  on  functionality  and 
consistency  with  "commercial  best  practices" 

DSB 

1994 

83 

Minimize  DoD  regulations  for  review  and 
documentation  that  are  different  than  "commercial 
best  practices" 

DSB 

1994 

84 

Provide  expertise  and  resource  to  ensure  coordinated 
DoD  participation  in  commercial/international 
standards  and  users  groups 

DSB 

1994 

85 

Provide  for  evolution  of  the  DoD  Software 

Technology  Strategy  to  align  with  the  emerging 
commercial  technology  and  practices 

DSB 

1994 

86 

Apply  Evolutionary  Development  with  rapid 
deployment  of  initial  functional  capability 

DSB 

1994 
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87 

Establish  mechanism  to  allow  both  current  ability  to 
perform  as  well  as  past  performance  key  factors  in 
source  selection 

-Require  source  selection  evaluation  of  development 
contractors  through  a  formal  software  process 
capability  evaluation 

DSB 

1994 

88 

Encourage  competition  of  technical  approach  vs.  cost 

DSB 

1994 

89 

Prior  to  RFP,  Government  should  perform 

Independent  Market  Analysis  of  off-the-shelf  and 
contractor  products  to  assure  "Best  Value" 

DSB 

1994 

90 

Require  trade  studies  and  analysis  of  the  use  of 

COTS  in  DoD's  software  acquisition  process  where 
effective 

Use  of  COTS  appropriate  when: 

-  Defining  Requirements 

-  Rapid  prototyping  situations 

-  Not  required  to  tailor  COTS  source  code  to 
application 

-  Not  required  to  be  error-free 

-  COTS  software  is  "Close  Enough"  to  tailor 
requirements 

DSB 

1994 

91 

Define  successful  performance  on  contracts  as 
delivering  solution  (with  predictable  Price,  Schedule, 
and  Functionality)  not  adherence  to  Government 
processes,  procedure  and  specifications 

DSB 

1994 

92 

Encourage  offers  to  demonstrate  as  much 
functionality  as  possible  as  part  of  bid  without 
eliminating  domain  knowledgeable  competition 

DSB 

1994 

93 

Establish  "Customer  Friendly"  application-specific 
information  technology  "Component  Stores" 

-  Generic  Architectures  for  Specific  domains 

-  Rapid  requirements  definition  process  and 
prototyping 

-  Reusable,  prequalified  components 

-  Assemble  systems  rather  than  develop  them 

-  Reduce  lead  time 

-  Security  is  not  paramount 

DSB 

1994 

94 

Capitalize  on  Innovative  Cost-Effective  techniques  for 
acquiring  and  using  COTS  software  products 
-  Such  as  use  of  enterprise  licenses 

DSB 

1994 

95 

Have  Program  Managers  manage  3  of  3 
(Price/Schedule/Functionality)  but  only  constrain  2 

DSB 

1994 

96 

Define  software  architecture  to  enable  rapid  changes 
and  reuse 

DSB 

1994 

97 

To  achieve  the  benefits  of  using  standards-based 
architectures,  DoD  must  manage  programs  using: 
-Early  systems  engineering 
-  Interactive  Development 
-Proactive  participation  in  development  of  these 
standards 

DSB 

1994 

51 


98 

Emphasize  use  of  software  architecture 

-  Establish  model  and  context  for  architecture  section 

-  Standards-based  with  emphasis  on 
"unimplemented" 

-  Require  vendors  to  propose,  manage,  and  control 
architecture 

-  Require  delivery  of  software  architecture  definition 
as  first  step  in  any  software  acquisition 

-Foster  migration  strategies  at  architecture  level 

DSB 

1994 

99 

Provide  government  funded  vehicle  in  contracts  to 
incentives  development  of  reusable  software 

DSB 

1994 

100 

Provide  incentives  and  guidelines  to  encourage 
software  reuse  (architecture-based  reuse) 

DSB 

1994 

101 

Promote  Development/Use  Community-wide  Metrics 
and  Models  (e.g.,  SEI's  Capability  Maturity  Model) 

DSB 

1994 

102 

Upgrade  educational  requirements  for  personnel 
assigned  to  acquisition,  management,  development, 
and  oversight  of  software  intensive  programs 

DSB 

1994 

103 

Establish  DoD-Wide  software  program  management 
education  and  training  initiative 

DSB 

1994 

104 

Change  DSMC  and  IRMC  courses  for  PMs  to  reflect 
best  commercial  practices  and  other 
recommendations  of  this  Task  Force  and  provide  for 
changes  to  reflect  the  dynamics  of  the  software 
industry 

DSB 

1994 

105 

Develop  and  provide  interactive  training  tools  for 
senior  managers  to  perfect  software  management 
skills 

DSB 

1994 

106 

Incorporate  software  management  principles  in  senior 
management  education  and  seminars  (including 
senior  service  colleges) 

DSB 

1994 

107 

Provide  mechanisms  for  keeping  software  expertise 
current  in  the  workplace 

DSB 

1994 

108 

Establish  norms  for  the  number  of  software  experts 
on  program  offices 

DSB 

1994 

109 

Develop  Acquisition  Managers  with  software  program 
management  expertise 

-  Integrate  software-qualified  personnel  into  senior 
acquisition  staff 

DSB 

1994 

110 

Develop  expertise  in  analysis  of  domain  software 
design 

-  Promote  software  reuse  in  the  design 

DSB 

1994 

111 

Have  program  managers  stay  with  programs  at  least 
through  Beta  testing  to  maintain  continuity  of 
understanding  of  original  nuances  of  requirements 

DSB 

1994 

112 

Rotate  government  and  contractor  personnel 
between  the  PM  and  development  organization  to 
build  understanding  and  trust;  encourage  use  of  IPA's 
form  industry 

DSB 

1994 

52 


113 

Require  early  interaction  between  user,  acquisition, 
agent  and  developer;  identify  and  get  early  user 
involvement 

DSB 

1994 

114 

Revise  procedures  encouraging  interaction  between 
user  and  developer  and  achieving  early  functionality 

DSB 

1994 

115 

Tailor  operational  testing  to  develop  DoD  "Beta  Test" 
philosophy 

-Allow  fielding  of  software  direct  form  test  beds  with 
user  consent 

DSB 

1994 

116 

Revise  Milestones  for  Software-Intensive 

Development 

-Address  the  need  for  software  first  philosophy 
-Provide  for  a  layered  software/hardware  standards 
based  architecture 

DSB 

1994 

117 

Require  planning  for  maintenance  at  beginning  of 
development  process 

DSB 

1994 

118 

Reduce  documentation  and  review  requirements  for 
"mature"  companies  (i.e.  Companies  determined  to 
be  "mature"  through  evaluation  mechanisms) 

DSB 

1994 

119 

Assign  responsibility  within  Government  for  domain 
analysis  and  product  line  developments 

DSB 

1994 

120 

Do  not  require  C-level  specifications  for  software 
projects  developed  in  Ada 

DSB 

1994 

Due  to  decreasing 
support  by  industry 
and  academia  the 
mandate  for  the 
required  use  of  Ada 
removed  and  the  Ada 
JPO  was  dissolved 

121 

Review  all  existing  military  standards  and  military 
specification  pertaining  to  software  development  and 
documentation,  for  continued  applicability,  such  as 
DOD-STD  2167 

DSB 

1994 

New  Policies, 

Standards,  and 
Directives  have  been 
implemented 

122 

Strengthen  technology  base 

DSB 

2000 

123 

Collect,  disseminate,  and  employ  best  practices 

DSB 

2000 

124 

Stress  software  past  performance  and  process 
maturity 

DSB 

2000 

125 

Restructure  contract  incentives 

DSB 

2000 

126 

Initiate  Independent  Expert  Reviews  (lERs) 

DSB 

2000 

127 

Improve  software  skills  of  acquisition  and  program 
management 

DSB 

2000 

53 


128 

To  strengthen  DLA,  Marine  Corps,  and  the  Navy 
software  systems  development,  acquisition,  an 
engineering  processes,  we  recommend  that  the 
secretary  of  Defense  direct  the  Director  DLA,  the 
Commandant  of  the  Marine  Corps,  and  the  Secretary 
of  the  Navy  to  establish  SPI  programs  where  this 
report  shows  none  currently  exist.  In  doing  so,  these 
officials  should  consider  following  the  best  practices 
embodied  in  the  SEI  IDEAL  model  and  drawing  form 
experiences  of  the  Army,  Air  Force,  DFAS,  and  some 
Navy  units 

GAO-0 1- 
116 

129 

To  strengthen  DoD-wide  SPI,  we  recommend  that  the 
Secretary  of  Defense  direct  the  Assistant  Secretary  of 
Defense  for  Command,  Control,  Communications, 
and  Intelligence,  in  collaboration  with  the  Under 
Secretary  of  defense  for  Acquisition,  Technology,  and 
Logistics,  to  (1)  issue  a  policy  requiring  DoD 
components  that  are  responsible  for 
systems/software  development,  acquisition,  or 
engineering  to  implement  SPI  programs,  and  (2) 
develop  and  issues  SPI  guidance  and,  in  doing  so, 
consider  basing  this  guidance  on  the  SEI  IDEAL 
model  and  the  positive  examples  of  SPI  within  the 
Army,  Air  Force,  DFAS,  and  some  Navy  units  cited  in 
this  report 

G  AO-01  - 
116 

130 

The  Secretary  of  Defense  should  direct  the  Assistant 
Secretary  for  Command,  Control,  Communications, 
and  Intelligence  to  (1)  annually  determine  the 
components'  compliance  with  the  SPI  policy  and  (2) 
establish  and  promote  a  means  for  sharing  SPI 
lessons  learned  and  best  practices  knowledge 
throughout  DoD 

GAO-0 1- 
116 

131 

To  ensure  that  DLA  has  in  place  the  necessary 
process  controls  to  acquire  quality  software 
consistently  on  future  acquisition  projects,  we 
recommend  that  the  Secretary  also  direct  DLA  to: 
issues  a  policy  requiring  that  (1)  DLA  software¬ 
intensive  acquisition  projects  satisfy  all  applicable  SEI 
SA-CMM  level-2  key  processes  areas  and  the  level-3 
risk  management  key  process  maturity  levels;  and 
direct  the  Chief  Information  Officer  (CIO)  to  establish 
and  sustain  a  software  process  improvement 
program,  including  (1)  developing  and  implementing 
a  software  process  improvement  plan  that  specifies 
measurable  goals  and  milestones,  (2)  prohibiting 
adequate  resources  to  the  program,  (3)  reporting  to 
the  Director  every  6  months  on  progress  against 
plans 

GAO-02- 

9 

54 


132 

To  reduce  the  software  acquisition  risks  associated 
with  its  two  ongoing  acquisition  projects,  we 
recommend  that  the  Secretary  of  Defense  direct  the 
Director  of  DLA  to  immediately  correct  each  BSM  and 
FAS  software  acquisition-practice  weakness 
identified  in  this  report 

GAO-02- 

9 

Specific  to  DLA 

133 

These  practices  should  be  included  and  enforced 
with  controls  and  incentives  in  DoD's  acquisition 
policy,  software  acquisition  improvement  plans  and 
development  contracts 

GAO-04- 

393 

134 

To  assure  DoD  appropriately  sets  and  manages 
requirements,  we  recommend  that  DoD  document 
that  software  requirements  are  achievable  based  on 
knowledge  obtained  form  systems  engineering  prior 
to  beginning  development  and  that  DoD  and  the 
contractor  have  a  mutual  understanding  of  the 
software  requirements.  Furthermore,  we  recommend 
that  trade-off  analyses  be  performed,  supported  by 
systems  engineering  analysis,  considering 
performance,  cost,  and  schedule  impacts  of  major 
changes  to  software  requirements 

GAO-04- 

393 

135 

The  ensure  DoD  acquisitions  are  managed  to 
disciplines  process,  acquires  should  develop  a  list  of 
systems  engineering  deliverable  (including  software), 
tailored  to  the  program  characteristics,  and  based  on 
the  results  of  systems  engineering  activities  that 
software  developers  are  required  to  provide  at  the 
appropriate  stages  of  the  systems  development 
phases  of  requirements,  design,  fabrication,  coding, 
integration,  and  testing 

GAO-04- 

393 

136 

To  ensure  DoD  has  the  knowledge  it  needs  to 
oversee  software-intensive  acquisitions,  we 
recommend  that  acquires  require  software 
contractors  to  collect  and  report  metrics  related  to 
cost,  schedule,  size,  requirements,  tests,  defects,  and 
quality  to  program  offices  on  a  monthly  basis  and 
before  program  milestones  and  that  acquirers  should 
ensure  contractors  have  an  earned  value 
management  system  that  reports  cost  and  schedule 
information  at  a  level  of  work  that  provides 
information  specific  to  software  development 

GAO-04- 

393 

137 

Develop  and  implement  an  explicit  plan  for 
incorporating  onto  the  5000  series  the  best  practices 
and  associated  activities  currently  missing  from  the 
series.  We  recommend  that  the  plan  specify  tasks  to 
be  performed,  resources  needed  and  assigned,  and 
milestones  for  completing  tasks. 

GAO-04- 

722 

55 


138 

To  ensure  that  the  best  practices  provided  for  in  DoD 
acquisition  policies  and  guidance  are  appropriately 
followed,  we  also  recommend  that  the  above 
recommended  plan  incorporate  steps  to  include  in 
DoD's  acquisition  policies  a  provision  for 
measurement  and  verification  of  best  practices. 

GAO-04- 

722 

139 

Investment  decisions  throughout  a  system's  life  cycle 
are  based  on  a  continuous  set  of  tradeoffs  among 
capabilities  available  in  commercial  components 
(current  and  future),  the  architecture  environment  in 
which  the  system  is  to  operate,  defined  systems 
requirements,  and  existing  cost/schedule  constraints 

GAO-04- 

722 

140 

Evaluation  criteria  are  established  for  selecting 
among  commercial  component  options  that  include 
both  defined  system  requirements  and 
vendor/commercial  product  characteristics 

GAO-04- 

722 

141 

Systems  integration  contractors  are  explicitly 
evaluated  on  their  ability  to  implement  commercial 
components 

GAO-04- 

722 

142 

Modification  of  commercial  components  is 
discouraged  dandy  allowed  only  if  justified  by  a 
thorough  analysis  of  life-cycle  costs  and  benefits 

GAO-04- 

722 

143 

Acquisition  plans  provide  for  preparing  users  for  the 
impact  that  the  business  processes  embedded  in  the 
commercial  components  will  have  on  their  respective 
roles  and  responsibilities 

GAO-04- 

722 

144 

Product  line  requirements-rather  than  just  the 
requirements  for  the  systems  being  acquired-are  an 
explicit  consideration  in  each  acquisition 

GAO-04- 

722 

145 

Acquisition  reviews  include  the  status  of  identified 
risks 

GAO-04- 

722 

146 

Acquisition  project  managers  activities  are 
communicated  to  all  stakeholders 

GAO-04- 

722 

147 

Modification  or  upgrades  to  deployed  versions  of 
systems  components  are  based  on  deliberate  and 
thorough  research,  analysis,  and  evaluation  of  the 
components  interdependencies 

GAO-04- 

722 

148 

Changes  affecting  how  users  will  be  expected  to  use 
the  system  to  execute  their  jobs  are  actively 
managed 

GAO-04- 

722 

149 

AFSC,  with  the  Joint  Logistics  Commanders,  should 
expedite  preparation  and  distribution  of  the  2168 
guide  book  and  support  maintained  of  this  and  other 
software  guidebooks  over  time 

NRC 

1989 

56 


150 

For  key  technologies  in  systems  and  application 
areas  where  operational  threats  or  requirements 
change  rapidly,  AFSC  should  fund  parallel  technology 
programs  at  the  systems  level  to  foster  a  ready 
industrial  base  from  which  to  compete  single  phase 
systems  acquisition 

NRC 

1989 

151 

AFSC  should  increase  its  technology  base 
investment  in  software  engineering  technology,  which 
is  currently  running  at  less  than  $8  million  per  year. 

This  increase  should  include  Air  Force  laboratories 
more  broadly  and  directly  than  in  the  past  decade.  As 
a  way  to  improve  software  technology  transfer,  and  in 
line  with  its  usual  strategy,  AFSC  should  select 
programs  for  application  and  demonstration  of 
advances  in  software  engineering  technology,  and 
provide  separate  6.3  funding  to  support 
demonstrations 

NRC 

1989 

152 

AFSC  should  consider  funding  a  program  to  evaluate 
candidate  SEEs  and  where  applicable,  stand-alone 
tools,  for  consideration  as  acceptable  environments 
and  tool  sets 

NRC 

1989 

153 

AFSC  should  require  the  use  of  commercial  off-the- 
shelf  software  test  technology  in  systems  and 
software  development,  make  it  a  part  of  the 
technology  and  software  process  research  and 
development  programs  to  further  advance  the  area, 
and  apply  it  throughout  the  software  life  cycle 

NRC 

1989 

154 

AFSC  should  create  and  fund  a  project  to  provide 
support  for  the  software  systems  engineering 
advisory  team(s)  of  recommendation  8,  in  particular 
to  capture  the  knowledge  gained  and  used  by  the 
team  members  for  use  via  knowledge-based  tools. 

This  could  be  a  valuable  lead  project  for  later  use  of 
similar  tools,  more  broadly  in  AFSC  systems  and 
software  acquisition  management 

NRC 

1989 

155 

AFSC  should  select  an  appropriate  program  (or 
programs)  through  which  to  implement  incremental 
acquisition,  using  it  (or  them)  to  articulate  to  the 

Office  of  Management  and  Budget  and  the  Congress 
the  need  for  and  special  benefits  of  an  evolutionary, 
incremental,  acquisition  process 

NRC 

1989 

156 

AFSC  should  take  steps  to  increase  the  motivation 
for  innovative  acquisition  tailoring.  AFSC  should  issue 
policy  a  statement,  conduct  workshops,  and  distribute 
guidebooks 

NRC 

1989 

57 


157 

AFSC  should  direct  its  product  division  to  tailor  the 
contract  form  for  each  specific  programs  needs;  in 
particular,  AFSC  should  avoid  using  firm  fixed  price 
contracts  for  unprecedented  programs  (This  will 
require  management  follow-up,  consistency,  and  the 
support  of  higher  authority) 

NRC 

1989 

158 

Product  divisions  should  be  directed  to  specify  use  of 
an  SEE  for  each  program  having,  as  an  example,  a 
software  staff  of  more  than  12  people,  and  to  require 
proof  of  its  existence  and  the  contractor's  knowledge 
of  its  effective  use,  in  order  to  qualify 

NRC 

1989 

159 

When  a  program  manager  is  faced  with  late 
identification  of  software  requirements  that  can  be 
deferred  to  a  later  time  or  capability  block,  AFSC 
management  guidance  should  encourage  and 
support  this  deferral  and  accept  the  consequences  of 
doing  so. 

NRC 

1989 

160 

AFSC  should  ensure  adequate  software  risk 
reduction  for  unprecedented  systems  during  a  full- 
scale  development.  For  unprecedented  systems, 

AFSC  should  provide  policy  guidance  for  competitive 
two-phased  procurements,  such  that  software  risks 
are  reduced  to  a  practical  minimum  before  proposal 
are  prepared. 

NRC 

1989 

161 

Each  program  involving  software  should  be  required 
to  carry  out  early  identification  of  critical  software 
issues  and  to  develop  and  maintain  a  Software  Risk 
Management  Plan. 

NRC 

1989 

162 

AFSC,  with  AFLC  and  the  using  commands,  should 
sponsor  a  fresh  look  are  actual  maintainer 
documentation  needs.  This  review  should  consider 
the  growing  automation  of  documentation  by 
contractors,  and  how  that  might  be  used  to  reduce 
the  cost  or  improve  utility  of  data 

NRC 

1989 

163 

Product  divisions  or  Headquarters  AFSC  should 
regularly  monitor  computer  resources  working  group 
performance.  Explicit  evaluations  should  be  solicited 
from  using  commands  and  AFLC 

NRC 

1989 

164 

AFSC  should  select  key  programs  that  have  high 
concerns  for  reliability,  maintainability,  re-usability, 
and  interoperability  for  demonstration  and  evaluation 
of  this  prototype  product  quality  assessment  scheme. 
AFSC  should  invest  funds  to  merge  product  and 
process  quality  measurement  schemes  to  get 
increased  benefits  and  to  keep  the  measurement 
technology  updated  to  the  needs  of  future  life  cycle 
models 

NRC 

1989 

58 


165 

AFSC  should  initiate  a  program  in  the  style  of 
MANTECH  (the  manufacturing  technology  program) 
to  transfer  software  development  process 
technologies  into  actual  minor  systems  and  software 
development  programs 

NRC 

1989 

166 

AFSC  special  management  of  software  skills  should 
include  a  software  systems  engineering  advisory 
team  and  special  career  tailoring  for  selected  officers 
and  civilians 

NRC 

1989 

167 

AFSC,  in  collaboration  with  others,  should  make 
available  to  officers  and  civilians  a  mid-career 
systems  engineering  and  software  engineering 
graduate  program  and  appropriate  short  course 

NRC 

1989 

168 

AFSC  should  broaden  the  base  of  its  personnel 
skilled  in  acquisition  of  software-intensive  systems; 
prepare,  use,  and  maintain,  current  guide  books;  and 
exercise  special  management  of  skilled  personnel 

NRC 

1989 

169 

User  involvement  should  be  tailored  for  each 
program,  varying  form  cases  requiring  very  limited 
involvement  to  ones  in  which  user  will  assume  lead 
role 

NRC 

1989 

170 

The  Air  Force  should  consider  revision  of  AFR  800- 
14  paragraph  5-3,  Test  Planning  ,  and  all  derived 
directives,  to  require  demonstration  of  testing  of  every 
instruction  within  the  software  prior  to  completion  of 
development,  test,  and  evaluation  (DT&E). 
Implementation  needs,  costs,  and  expected  benefits 
should  be  analyzed  by  experts  prior  to  implementing 
revisions 

NRC 

1989 

171 

Each  program  should  consider  using  the  designated 
software  "maintainer"  (operational  phase)  as  the 
independent  validation  and  verification  agent  during 
software  development 

NRC 

1989 

172 

AFSC,  working  with  the  Joint  Logistics  Commanders 
organization,  should  ensure  that  development  models 
and  accompanying  rational  alternatives  to  the 
waterfall  model,  based  on  risk  reduction  concepts, 
are  included  in  forthcoming  Handbook  287  for  DoD- 
STD-2167A,  with  supporting  direction  in  AFR  800-2 
and  800-14 

NRC 

1989 

173 

AFSC  must  strongly  encourage  AFLC  and  the  using 
commands  toward  collected  support  for  software  in 
integrated  systems,  rather  than  complex 
reprogramming  without  adequate  resources  in  the 
field 

NRC 

1989 

Specific  to  AFSC 

59 


174 

When  software  development  contracts  are  granted  to 

NRC 

OBE-  Technology 

design  groups  that  are  organizationally  or 
geographically  separated  ,  near-term  management 
criteria  in  source  selection  should  emphasize  use  of 
modern  telecommunications  and  division  of  tasks  to 
reduce  requirements  for  interface  among  separate 
locations  or  organizations. 

1989 

today  exists  to 
accomplish  this  task 

60 


Appendix  C.  Question  Traceability 


Question 

Number 

Question 

Recommendation 

Source 

1 

Is  there  a  clear  set  of 
Acquisition  Policy  for 
software  development? 

A.  Is  there  enough  up-to-date 
guidance  for  accomplishing 
your  program? 

SS-3 1 1  Establish  clear 

Acquisition  Policy  for  Software 

DSB 

1989 

1 

AFSC,  with  the  Joint  Logistics 
Commanders,  should  expedite 
preparation  and  distribution  of  the 
2168  guide  book  and  support 
maintained  of  this  and  other 
software  guidebooks  over  time 

NRC 

1989 

2 

Have  you  seen  impacts  from 
Section  804? 

To  strengthen  DLA,  Marine 

Corps,  and  the  Navy  software 
systems  development,  acquisition, 
an  engineering  processes,  we 
recommend  that  the  secretary  of 
Defense  direct  the  Director  DLA, 
the  Commandant  of  the  Marine 
Corps,  and  the  Secretary  of  the 
Navy  to  establish  SPI  programs 
where  this  report  shows  none 
currently  exist.  In  doing  so,  these 
officials  should  consider 
following  the  best  practices 
embodied  in  the  SEI  IDEAL 
model  and  drawing  form 
experiences  of  the  Army,  Air 

Force,  DFAS,  and  some  Navy 
units 

GAO- 

01-116 

2 

SS-3 16  Enhance  Interaction 
between  Activities 

DSB 

1989 

61 


Question 

Number 

Question 

Recommendation 

Source 

2 

The  Secretary  of  Defense  should 
direct  the  Assistant  Secretary  for 
Command,  Control, 
Communications,  and  Intelligence 
to  (1)  annually  determine  the 
components'  compliance  with  the 
SPI  policy  and  (2)  establish  and 
promote  a  means  for  sharing  SPI 
lessons  learned  and  best  practices 
knowledge  throughout  DoD 

GAO- 

01-116 

2 

To  strengthen  DoD-wide  SPI,  we 
recommend  that  the  Secretary  of 
Defense  direct  the  Assistant 
Secretary  of  Defense  for 

Command,  Control, 
Communications,  and 

Intelligence,  in  collaboration  with 
the  Under  Secretary  of  defense 
for  Acquisition,  Technology,  and 
Logistics,  to  (1)  issue  a  policy 
requiring  DoD  components  that 
are  responsible  for 
systems/software  development, 
acquisition,  or  engineering  to 
implement  SPI  programs,  and  (2) 
develop  and  issues  SPI  guidance 
and,  in  doing  so,  consider  basing 
this  guidance  on  the  SEI  IDEAL 
model  and  the  positive  examples 
of  SPI  within  the  Army,  Air 

Force,  DFAS,  and  some  Navy 
units  cited  in  this  report 

GAO- 

01-116 

62 


Question 

Number 

Question 

Recommendation 

Source 

3 

It  has  been  suggested  that  the 
commercial  market,  not  the 
DoD,  is  clearly  driving 
today’s  infonnation 
technology.  DoD,  however, 
must  stay  abreast  of  the  most 
current  technology,  and  are 
there  areas  that  are  imperative 
to  the  success  of  DoD  which 
are  not  being  cover  by  the 
commercial  market? 

Emphasize  Technology  Transfer 
(External  and  Internal) 

-  Fund  technology  transfer 
programs 

-  Initiate  demonstration  program 
(e.g.,  ATDs)  to  facilitate  software 
technology  insertion  into  systems. 
Examples  of  candidate  criteria: 

-  Open  Standards,  Use  of  COTS 
and  GOTS,  Frequent  releases  to 
include  numbers  of  users, 
multiple  platforms,  and  stratifies 
commercial  standards  and 
interoperability  standards  across 
DoD  organization 

DSB 

1994 

3 

For  key  technologies  in  systems 
and  application  areas  where 
operational  threats  or 
requirements  change  rapidly, 

AFSC  should  fund  parallel 
technology  programs  at  the 
systems  level  to  foster  a  ready 
industrial  base  from  which  to 
compete  single  phase  systems 
acquisition 

NRC 

1989 

3 

Strengthen  technology  base 

DSB 

2000 

4 

In  addition  to  looking  at 

COTS  products  should  we 
continue  to  be  looking  for 
opportunities  to  buy,  in  the 
civilian  market,  tools, 
methods,  environments,  and 
application  software? 

AFSC  should  consider  funding  a 
program  to  evaluate  candidate 

SEEs  and  where  applicable, 
stand-alone  tools,  for 
consideration  as  acceptable 
environments  and  tool  sets 

NRC 

1989 

4 

All  the  methodological  efforts, 
especially  STARS,  should  look  to 
see  how  commercially  available 
software  tools  can  be  selected  and 
standardized  for  DoD  needs 

DSB 

1987 

63 


Question 

Number 

Question 

Recommendation 

Source 

4 

AFSC  should  require  the  use  of 
commercial  off-the-shelf  software 
test  technology  in  systems  and 
software  development,  make  it  a 
part  of  the  technology  and 
software  process  research  and 
development  programs  to  further 
advance  the  area,  and  apply  it 
throughout  the  software  life  cycle 

NRC 

1989 

4 

Increase  tech  base  funding  for 
security  audit  tools  for  systems 
employing  COTS 

DSB 

1994 

5 

In  your  experience  has  the 

DoD  been  effective  at 
collecting  and  disseminating 
best  practices  of  both  the 
government  and  industry? 

Collect,  disseminate,  and  employ 
best  practices 

DSB 

2000 

5 

AFSC  should  create  and  fund  a 
project  to  provide  support  for  the 
software  systems  engineering 
advisory  team(s)  of 
recommendation  8,  in  particular 
to  capture  the  knowledge  gained 
and  used  by  the  team  members 
for  use  via  knowledge-based 
tools.  This  could  be  a  valuable 
lead  project  for  later  use  of 
similar  tools,  more  broadly  in 
AFSC  systems  and  software 
acquisition  management 

NRC 

1989 

5 

DoD  should  examine  and  revise 
regulations  to  approach  modern 
commercial  practices  insofar  as 
practicable  and  appropriate. 

DSB 

1987 

5 

Establish  acquisition  focus  on 
functionality  and  consistency  with 
"commercial  best  practices" 

DSB 

1994 

5 

Minimize  DoD  regulations  for 
review  and  documentation  that 
are  different  than  "commercial 
best  practices" 

DSB 

1994 

64 


Question 

Number 

Question 

Recommendation 

Source 

5 

Provide  expertise  and  resource  to 
ensure  coordinated  DoD 
participation  in 
commercial/international 
standards  and  users  groups 

DSB 

1994 

5 

Provide  for  evolution  of  the  DoD 
Software  Technology  Strategy  to 
align  with  the  emerging 
commercial  technology  and 
practices 

DSB 

1994 

5 

Develop  and  implement  an 
explicit  plan  for  incorporating 
onto  the  5000  series  the  best 
practices  and  associated  activities 
currently  missing  from  the  series. 
We  recommend  that  the  plan 
specify  tasks  to  be  performed, 
resources  needed  and  assigned, 
and  milestones  for  completing 
tasks. 

GAO- 

04-722 

5 

Direct  STARS  to  choose  several 
real  programs  early  in 
development  and  augment  their 
funding  to  ensure  the  use  of 
existing  practices  and  tools. 

DSB 

1987 

5 

SS-315  Develop  a  Computer 
Resource  Data  Base 

DSB 

1989 

5 

These  practices  should  be 
included  and  enforced  with 
controls  and  incentives  in  DoD's 
acquisition  policy,  software 
acquisition  improvement  plans 
and  development  contracts 

GAO- 

04-393 

6 

Is  there  a  process  to  evaluate  the 
usage  of  best  practices? 

To  ensure  that  the  best  practices 
provided  for  in  DoD  acquisition 
policies  and  guidance  are 
appropriately  followed,  we  also 
recommend  that  the  above 
recommended  plan  incorporate  steps 
to  include  in  DoD's  acquisition 
policies  a  provision  for  measurement 
and  verification  of  best  practices. 

GAO- 

04-722 

65 


Question 

Number 

Question 

Recommendation 

Source 

7 

Is  it  appropriate  to  use 
Evolutionary  Acquisition, 
including  simulation  and 
prototyping,  for  software 
development? 

Should  it  be  the  primary 
model? 

Use  evolutionary  acquisition, 
including  simulation  and 
prototyping,  as  discussed  else 
ware  in  this  report,  to  reduce  risk. 

DSB 

1987 

7 

Apply  Evolutionary  Development 
with  rapid  deployment  of  initial 
functional  capability 

DSB 

1994 

7 

AFSC  should  select  an 
appropriate  program  (or 
programs)  through  which  to 
implement  incremental 
acquisition,  using  it  (or  them)  to 
articulate  to  the  Office  of 
Management  and  Budget  and  the 
Congress  the  need  for  and  special 
benefits  of  an  evolutionary, 
incremental,  acquisition  process 

NRC 

1989 

7 

The  Undersecretary  of  Defense 
(Acquisition)  should  update  DoD 
Directive  5000.29,  "Management 
of  Computer  Resources  in  Major 
Defense  Systems",  so  that  it 
mandates  the  iterative  setting  of 
specifications,  and  rapid 
prototyping  of  specified  systems 
and,  incremental  development. 

DSB 

1987 

7 

DoD  STD  2167  should  be  further 
revised  to  remove  any  remaining 
dependence  upon  the  assumption 
of  the  "waterfall"  model  and 
institutionalize  rapid  prototyping 
and  incremental  development. 

DSB 

1987 

7 

SS-1 14  Evaluate  Software  Life 
Cycle  Models 

DSB 

1989 

7 

Each  service  should  provide  its 
software  Product  Development 
Division  with  the  ability  to  do 
rapid  prototyping  in  conjunction 
with  users. 

DSB 

1987 

66 


Question 

Number 

Question 

Recommendation 

Source 

7 

AFSC,  working  with  the  Joint 
Logistics  Commanders 
organization,  should  ensure  that 
development  models  and 
accompanying  rational 
alternatives  to  the  waterfall 
model,  based  on  risk  reduction 
concepts,  are  included  in 
forthcoming  Handbook  287  for 
DoD-STD-2 1 67A,  with 
supporting  direction  in  AFR  800- 
2  and  800-14 

NRC 

1989 

8 

Should  the  Software 
Acquisition  Process  be 
standardized  or  tailored  for 
each  system? 

AFSC  should  take  steps  to 
increase  the  motivation  for 
innovative  acquisition  tailoring. 
AFSC  should  issue  policy  a 
statement,  conduct  workshops, 
and  distribute  guidebooks 

NRC 

1989 

8 

A.  Based  on  classifications 
such  as:  Life  cycle  Model, 
Requirement  Stability,  Reuse 
potential,  Contract  and 

Support  Strategy,  and  %  of 
new  development,  COTS, 
Modified  COTS,  or  Custom? 

SS-133  Tailor  Software 

Acquisition  Process  to  Systems 

DSB 

1989 

8 

B.  Should  programs  be 
tailored  based  on  amount  of 
user  involvement? 

The  Undersecretary  of  Defense 
(Acquisition)  should  adopt  a  four 
category  classification  as  a  bias 
for  acquisition  policy:  Standard, 
Extended,  Embedded,  and 
Advanced. 

DSB 

1987 

8 

C.  Is  there  policy/guidance  on 
how  this  should  be  done? 

The  Undersecretary  of  Defense 
(Acquisition)  should  develop 
acquisition  policy,  procedures, 
and  guidance  for  each  category 
(Follow  on  to  above 
recommendation) 

DSB 

1987 

67 


Question 

Number 

Question 

Recommendation 

Source 

8 

AFSC  should  direct  its  product 
division  to  tailor  the  contract  form 
for  each  specific  programs  needs; 
in  particular,  AFSC  should  avoid 
using  firm  fixed  price  contracts 
for  unprecedented  programs  (This 
will  require  management  follow¬ 
up,  consistency,  and  the  support 
of  higher  authority) 

NRC 

1989 

9 

To  ensure  that  DLA  has  in  place 
the  necessary  process  controls  to 
acquire  quality  software 
consistently  on  future  acquisition 
projects,  we  recommend  that  the 
Secretary  also  direct  DLA  to: 
issues  a  policy  requiring  that  (1) 
DLA  software-intensive 
acquisition  projects  satisfy  all 
applicable  SEI  SA-CMM  level-2 
key  processes  areas  and  the  level- 
3  risk  management  key  process 
maturity  levels;  and  direct  the 

Chief  Information  Officer  (CIO) 
to  establish  and  sustain  a  software 
process  improvement  program, 
including  (1)  developing  and 
implementing  a  software  process 
improvement  plan  that  specifies 
measurable  goals  and  milestones, 
(2)  prohibiting  adequate  resources 
to  the  program,  (3)  reporting  to 
the  Director  every  6  months  on 
progress  against  plans 

GAO- 

02-9 

9 

Establish  mechanism  to  allow 
both  current  ability  to  perform  as 
well  as  past  performance  key 
factors  in  source  selection 
-Require  source  selection 
evaluation  of  development 
contractors  through  a  formal 
software  process  capability 
evaluation 

DSB 

1994 

68 


Question 

Number 

Question 

Recommendation 

Source 

9 

SS-134  Develop  a  Consistent 
Contracting  Approach 

DSB 

1989 

9 

Product  divisions  should  be 
directed  to  specify  use  of  an  SEE 
for  each  program  having,  as  an 
example,  a  software  staff  of  more 
than  12  people,  and  to  require 
proof  of  its  existence  and  the 
contractor's  knowledge  of  its 
effective  use,  in  order  to  qualify 

NRC 

1989 

9 

Reduce  documentation  and 
review  requirements  for  "mature" 
companies  (i.e.  Companies 
determined  to  be  "mature" 
through  evaluation  mechanisms) 

DSB 

1994 

10 

Is  evaluating  competitors  on 
their  technical  approach  rather 
than  cost,  feasible? 

Encourage  competition  of 
technical  approach  vs.  cost 

DSB 

1994 

11 

Is  requiring  the  Government, 
prior  to  RFP,  to  perfonn  an 
Independent  Market  Analysis 
of  Off-the-Shelf  and 
contractor  products  to  assure  a 
“Best  Value”  solution 
beneficial  to  the  acquisition  of 
software? 

Prior  to  RFP,  Government  should 
perfonn  Independent  Market 
Analysis  of  off-the-shelf  and 
contractor  products  to  assure 
"Best  Value" 

DSB 

1994 

11 

Require  trade  studies  and  analysis 
of  the  use  of  COTS  in  DoD's 
software  acquisition  process 
where  effective 

Use  of  COTS  appropriate  when: 

-  Defining  Requirements 

-  Rapid  prototyping  situations 

-  Not  required  to  tailor  COTS 
source  code  to  application 

-  Not  required  to  be  error-free 

-  COTS  software  is  "Close 

Enough"  to  tailor  requirements 

DSB 

1994 

69 


Question 

Number 

Question 

Recommendation 

Source 

11 

Investment  decisions  throughout 
a  system's  life  cycle  are  based  on 
a  continuous  set  of  tradeoffs 
among  capabilities  available  in 
commercial  components  (current 
and  future),  the  architecture 
environment  in  which  the  system 
is  to  operate,  defined  systems 
requirements,  and  existing 
cost/schedule  constraints 

GAO- 

04-722 

11 

Evaluation  criteria  are  established 
for  selecting  among  commercial 
component  options  that  include 
both  defined  system  requirements 
and  vendor/commercial  product 
characteristics 

GAO- 

04-722 

11 

Define  successful  performance  on 
contracts  as  delivering  solution 
(with  predictable  Price,  Schedule, 
and  Functionality)  not  adherence 
to  Government  processes, 
procedure  and  specifications 

DSB 

1994 

12 

When  considering  offers  in 
source  selection  process 
should  the  offers  be 
encouraged  to  demonstrate  as 
much  functionality  as 
possible? 

Encourage  offers  to  demonstrate 
as  much  functionality  as  possible 
as  part  of  bid  without  eliminating 
domain  knowledgeable 
competition 

DSB 

1994 

12 

How  was  experience  with 
COTS  usage  considered  in 
source  selection? 

Stress  software  past  performance 
and  process  maturity 

DSB 

2000 

12 

Systems  integration  contractors 
are  explicitly  evaluated  on  their 
ability  to  implement  commercial 
components 

GAO- 

04-722 

13 

Do  you  have  a  system  to  help 
identify  potential  COTS 
products  to  meet  systems 
requirements? 

SS-1 12  Develop  an  Approach  to 
Software  Reuse 

DSB 

1989 

70 


Question 

Number 

Question 

Recommendation 

Source 

13 

Establish  "Customer  Friendly" 
application-specific  infonnation 
technology  "Component  Stores" 

-  Generic  Architectures  for 

Specific  domains 

-  Rapid  requirements  definition 
process  and  prototyping 

-  Reusable,  prequaliified 
components 

-  Assemble  systems  rather  than 
develop  them 

-  Reduce  lead  time 

-  Security  is  not  paramount 

DSB 

1994 

14 

Should  Program  Managers 
assume  that  system  software 
requirements  can  be  met  with 
off-the-shelf  subsystems  and 
components  until  it  is  proved 
that  they  are  unique? 

The  Undersecretary  of  Defense 
(Acquisition)  and  the  Assistant 
Secretary  of  Defense 
(Comptroller)  should  direct 
Program  Managers  to  assume  that 
systems  software  requirements 
can  be  met  with  off-the-shelf 
subsystems  and  components  until 
it  is  proven  that  they  are  unique. 

DSB 

1987 

14 

Capitalize  on  Innovative  Cost- 
Effective  techniques  for  acquiring 
and  using  COTS  software 
products 

-  Such  as  use  of  enterprise 
licenses 

DSB 

1994 

15 

Should  the  modification  of 
commercial  components  be 
discouraged  and  allowed  only 
if  justified  by  a  thorough 
analysis  of  life-cycle  costs  and 
benefits? 

Modification  of  commercial 
components  is  discouraged  dandy 
allowed  only  if  justified  by  a 
thorough  analysis  of  life-cycle 
costs  and  benefits 

GAO- 

04-722 

15 

Acquisition  plans  provide  for 
preparing  users  for  the  impact  that 
the  business  processes  embedded 
in  the  commercial  components 
will  have  on  their  respective  roles 
and  responsibilities 

GAO- 

04-722 

71 


Question 

Number 

Question 

Recommendation 

Source 

16 

Is  it  beneficial  to  have 
program  managers  manage  3 
of  3 

(Price/Schedule/Functionality) 
but  only  constrain  2  of  3? 

Is  this  a  realistic  approach? 

Have  Program  Managers  manage 

3  of  3 

(Price/Schedule/Functionality) 
but  only  constrain  2  of  3 

DSB 

1994 

17 

How  do  you  know  your 
software  requirements  are 
feasible? 

How  do  you  know  you  and 
your  contractor  have  the  same 
understanding  of  the 
requirements? 

To  assure  DoD  appropriately  sets 
and  manages  requirements,  we 
recommend  that  DoD  document 
that  software  requirements  are 
achievable  based  on  knowledge 
obtained  form  systems 
engineering  prior  to  beginning 
development  and  that  DoD  and 
the  contractor  have  a  mutual 
understanding  of  the  software 
requirements.  Furthermore,  we 
recommend  that  trade-off 
analyses  be  performed,  supported 
by  systems  engineering  analysis, 
considering  perfonnance,  cost, 
and  schedule  impacts  of  major 
changes  to  software  requirements 

GAO- 

04-393 

17 

Was  a  trade-off  analyses 
performed,  supported  by 
systems  engineering  analysis, 
considering  perfonnance, 
cost,  and  schedule  impacts  of 
major  changes  to  software 
requirements? 

Product  line  requirements-rather 
than  just  the  requirements  for  the 
systems  being  acquired-are  an 
explicit  consideration  in  each 
acquisition 

GAO- 

04-722 

17 

SS-123  Establish  mechanism  for 
Reverse  Engineering 

DSB 

1989 

18 

Who  should  have  the 
authority  to  defer 
requirements? 

When  a  program  manager  is  faced 
with  late  identification  of 
software  requirements  that  can  be 
deferred  to  a  later  time  or 
capability  block,  AFSC 
management  guidance  should 
encourage  and  support  this 
deferral  and  accept  the 
consequences  of  doing  so. 

NRC 

1989 

72 


Question 

Number 

Question 

Recommendation 

Source 

18 

The  Undersecretary  of  Defense 
(Acquisitions)  and  the  Assistant 
Secretary  of  Defense 
(Comptroller)  should  by  directive 
spell  out  the  role  of  Using 
Commands  in  the  evolutionary 
and  incremental  development  of 
software  systems. 

DSB 

1987 

19 

What  is  the  role  of  a  software 
architecture  in  your  program? 
Has  it  improved  your  software 
development? 

Define  software  architecture  to 
enable  rapid  changes  and  reuse 

DSB 

1994 

19 

To  achieve  the  benefits  of  using 
standards-based  architectures, 

DoD  must  manage  programs 
using: 

-Early  systems  engineering 
-  Interactive  Development 
-Proactive  participation  in 
development  of  these  standards 

DSB 

1994 

19 

Emphasize  use  of  software 
architecture 

-  Establish  model  and  context  for 
architecture  section 

-  Standards-based  with  emphasis 
on  "unimplemented" 

-  Require  vendors  to  propose, 
manage,  and  control  architecture 

-  Require  delivery  of  software 
architecture  definition  as  first  step 
in  any  software  acquisition 
-Foster  migration  strategies  at 
architecture  level 

DSB 

1994 

20 

Do  you  use  incentives  on 
contract  for  the  contractor  to 
build  better  software? 

Should  there  be  incentives  for 
quality,  reuses,  and 
application  of  commercial 
best  practices? 

DoD  should  devise  increased 
profit  incentives  on  software 
quality. 

DSB 

1987 

73 


Question 

Number 

Question 

Recommendation 

Source 

20 

The  Undersecretary  of  Defense 
(Acquisition)  and  the  Assistant 
Secretary  of  Defense 
(Comptroller)  should  direct 
Program  Managers  to  identify  in 
their  programs  those  subsystems, 
components  and  perhaps  even 
modules,  that  may  be  expected  to 
be  acquired  rather  than  built;  and 
to  reward  such  acquisition  in  the 
RFP's. 

DSB 

1987 

20 

Restructure  contract  incentives 

DSB 

2000 

20 

The  Undersecretary  of  Defense 
(Acquisition)  should  develop 
economic  incentives,  to  be 
incorporated  into  standard 
contracts,  to  allow  contractors  to 
profit  from  offering  modules  for 
reuse,  even  though  built  with 

DoD  funds. 

DSB 

1987 

20 

The  Undersecretary  of  Defense 
(Acquisition)  should  develop 
economic  incentives,  to  be 
incorporated  into  all  cost-plus 
standard  contracts,  to  encourage 
contractors  to  buy  modules  and 
use  them  rather  than  building  new 
ones. 

DSB 

1987 

20 

DoD  should  devise  increased 
productivity  incentives  for 
custom-built  software  contracts, 
and  make  such  incentives 
contracts  the  standard  practice. 

DSB 

1987 

20 

Provide  government  funded 
vehicle  in  contracts  to  incentives 
development  of  reusable  software 

DSB 

1994 

20 

Provide  incentives  and  guidelines 
to  encourage  software  reuse 
(architecture -based  reuse) 

DSB 

1994 

20 

SS-321  Integrate  Software 

Quality  into  Process 

DSB 

1989 

74 


Question 

Number 

Question 

Recommendation 

Source 

21 

Is  there  a  standard  cost 
estimation  model  used  for 
your  program? 

SS-411  Enforce  Standard 

Software  Cost  Model 

DSB 

1989 

22 

Does  your  program  track 
actual  software  cost 
throughout  the  entire 
lifecycle? 

SS-413  Identify  and  Capture 

Actual  Software  Costs 

DSB 

1989 

23 

Do  you  have  an  established 
Risk  Management  Plan? 

Is  there  policy  on  how  a  RMP 
should  be  created? 

When  are  program  risks 
reviewed? 

AFSC  should  ensure  adequate 
software  risk  reduction  for 
unprecedented  systems  during  a 
full-scale  development.  For 
unprecedented  systems,  AFSC 
should  provide  policy  guidance 
for  competitive  two-phased 
procurements,  such  that  software 
risks  are  reduced  to  a  practical 
minimum  before  proposal  are 
prepared. 

NRC 

1989 

23 

Directive  5000.29  and  STD  2167 
should  be  revised  or  superseded 
by  policy  mandate  risk 
management  techniques  in 
software  acquisition,  as 
recommended  in  1983 

USAF/SAB  Study. 

DSB 

1987 

23 

Each  program  involving  software 
should  be  required  to  carry  out 
early  identification  of  critical 
software  issues  and  to  develop 
and  maintain  a  Software  Risk 
Management  Plan. 

NRC 

1989 

23 

Acquisition  reviews  include  the 
status  of  identified  risks 

GAO- 

04-722 
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Question 

Number 

Question 

Recommendation 

Source 

24 

How  do/did  you  decide  what 
software  engineering 
deliverables  to  require? 

The  ensure  DoD  acquisitions  are 
managed  to  disciplines  process, 
acquires  should  develop  a  list  of 
systems  engineering  deliverable 
(including  software),  tailored  to 
the  program  characteristics,  and 
based  on  the  results  of  systems 
engineering  activities  that 
software  developers  are  required 
to  provide  at  the  appropriate 
stages  of  the  systems 
development  phases  of 
requirements,  design,  fabrication, 
coding,  integration,  and  testing 

GAO- 

04-393 

24 

AFSC,  with  AFLC  and  the  using 
commands,  should  sponsor  a 
fresh  look  are  actual  maintainer 
documentation  needs.  This  review 
should  consider  the  growing 
automation  of  documentation  by 
contractors,  and  how  that  might 
be  used  to  reduce  the  cost  or 
improve  utility  of  data 

NRC 

1989 

25 

Does  your  program  have  a 
CRWG  or  similar  IPT?  How 
is  the  performance  of  that 
group  evaluated? 

Product  divisions  or  Headquarters 
AFSC  should  regularly  monitor 
computer  resources  working 
group  perfonnance.  Explicit 
evaluations  should  be  solicited 
from  using  commands  and  AFFC 

NRC 

1989 

26 

Have  you  had  an  Independent 
Expert  Review  (IER)? 

Initiate  Independent  Expert 
Reviews  (IERs) 

DSB 

2000 

27 

DoD  should  develop  metrics  and 
measuring  techniques  for 
software  quality  and 
completeness,  incorporate  these 
routinely  in  contracts. 

DSB 

1987 

27 

SS-326  Provide  Software 

Maturity  Management 

DSB 

1989 

76 


Question 

Number 

Question 

Recommendation 

Source 

27 

AFSC  should  select  key  programs 
that  have  high  concerns  for 
reliability,  maintainability,  re¬ 
usability,  and  interoperability  for 
demonstration  and  evaluation  of 
this  prototype  product  quality 
assessment  scheme.  AFSC  should 
invest  funds  to  merge  product  and 
process  quality  measurement 
schemes  to  get  increased  benefits 
and  to  keep  the  measurement 
technology  updated  to  the  needs 
of  future  life  cycle  models 

NRC 

1989 

28 

Would  the  use  of  standardized 
Software  maturity  metrics  be 
beneficial  in  the  development 
of  software? 

SS-1 13  Develop  and  Evaluate 
Software  Metrics 

DSB 

1989 

28 

Promote  Development/Use 
Community-wide  Metrics  and 
Models  (e.g.,  SEI's  Capability 
Maturity  Model) 

DSB 

1994 

28 

SS-1 1 1  Implement  an  Effective 
Software  R&D  Strategy 

DSB 

1989 

28 

SS-131  Develop  a  Strategy  for 
Technology  Insertion 

DSB 

1989 

28 

Focus  a  critical  mass  of  software 
research  effort  on  software  needs 
that  are  unique  to  SDI  objectives. 

DSB 

1987 

28 

AFSC  should  initiate  a  program 
in  the  style  of  MANTECH  (the 
manufacturing  technology 
program)  to  transfer  software 
development  process  technologies 
into  actual  minor  systems  and 
software  development  programs 

NRC 

1989 

28 

Task  the  new  STARS  director  to 
define  a  new  set  of  program  goals 
together  with  an  implementation 
plan;  emphasis  should  be  on 
visible,  early  milestones  that  have 
demonstratable  results 

DSB 

1987 

77 


Question 

Number 

Question 

Recommendation 

Source 

29 

How  do  you  measure  success 
in  your  program? 

DoD  should  develop  metrics  to 
measure  implementation  progress. 

DSB 

1987 

30 

Should  the  DoD  ensure  that 
contractors  have  an  earned 
value  management  system 
that  reports  cost  and  schedule 
information  at  a  level  of  work 
that  provides  information 
specific  to  software 
development? 

To  ensure  DoD  has  the 
knowledge  it  needs  to  oversee 
software-intensive  acquisitions, 
we  recommend  that  acquires 
require  software  contractors  to 
collect  and  report  metrics  related 
to  cost,  schedule,  size, 
requirements,  tests,  defects,  and 
quality  to  program  offices  on  a 
monthly  basis  and  before  program 
milestones  and  that  acquirers 
should  ensure  contractors  have  an 
earned  value  management  system 
that  reports  cost  and  schedule 
infonnation  at  a  level  of  work  that 
provides  infonnation  specific  to 
software  development 

GAO- 

04-393 

31 

Do  you  have  the  appropriate 
number  of  software  personnel 
with  the  right  skills  for  your 
program? 

SS-431  Develop  Software 
Engineering  Career  Program 

DSB 

1989 

31 

SS-223  Organize  to  Grow 

Software  Engineers 

DSB 

1989 

31 

SS-232  Develop  Operational 
Software  Literacy  Program 

DSB 

1989 

31 

SS-432  Improve  Incentives  For 
Military  Software  Experts 

DSB 

1989 

31 

SS-433  Establish  Career 
Subprogram  Management 

DSB 

1989 

31 

SS-434  Provide  Job  Challenge  for 
Software  Engineers 

DSB 

1989 

31 

Upgrade  educational 
requirements  for  personnel 
assigned  to  acquisition, 
management,  development,  and 
oversight  of  software  intensive 
programs 

DSB 

1994 

78 


Question 

Number 

Question 

Recommendation 

Source 

31 

AFSC  special  management  of 
software  skills  should  include  a 
software  systems  engineering 
advisory  team  and  special  career 
tailoring  for  selected  officers  and 
civilians 

NRC 

1989 

31 

AFSC,  in  collaboration  with 
others,  should  make  available  to 
officers  and  civilians  a  mid-career 
systems  engineering  and  software 
engineering  graduate  program  and 
appropriate  short  course 

NRC 

1989 

31 

Establish  DoD-Wide  software 
program  management  education 
and  training  initiative 

DSB 

1994 

31 

Change  DSMC  and  IRMC 
courses  for  PMs  to  reflect  best 
commercial  practices  and  other 
recommendations  of  this  Task 
Force  and  provide  for  changes  to 
reflect  the  dynamics  of  the 
software  industry 

DSB 

1994 

31 

Develop  and  provide  interactive 
training  tools  for  senior  managers 
to  perfect  software  management 
skills 

DSB 

1994 

31 

Incorporate  software  management 
principles  in  senior  management 
education  and  seminars  (including 
senior  service  colleges) 

DSB 

1994 

31 

Provide  mechanisms  for  keeping 
software  expertise  current  in  the 
workplace 

DSB 

1994 

31 

Do  you  have  the  appropriate 
number  of  software  personnel 
with  the  right  skills  for  your 
program? 

Establish  norms  for  the  number  of 
software  experts  on  program 
offices 

DSB 

1994 
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Question 

Number 

Question 

Recommendation 

Source 

31 

Develop  Acquisition  Managers 
with  software  program 
management  expertise 
-  Integrate  software-qualified 
personnel  into  senior  acquisition 
staff 

DSB 

1994 

31 

Develop  expertise  in  analysis  of 
domain  software  design 
-  Promote  software  reuse  in  the 
design 

DSB 

1994 

31 

AFSC  should  broaden  the  base  of 
its  personnel  skilled  in 
acquisition  of  software-intensive 
systems;  prepare,  use,  and 
maintain,  current  guide  books; 
and  exercise  special  management 
of  skilled  personnel 

NRC 

1989 

32 

Do  you  have  enough  in-house 
(within  SPO)  software 
expertise  or  do  you  rely  on 
center  engineering  staff  or 
contract  out  for  software 
expertise? 

Improve  software  skills  of 
acquisition  and  program 
management 

DSB 

2000 

32 

SS-221  Provide  One-Stop 

Support  for  Program  Managers 

DSB 

1989 

33 

Should  the  DOD  sharply 
reduce  in-house  software 
construction,  extension,  and 
maintenance,  limiting  such  to 
critical  functions  at 
operational  bases,  adaptation 
of  existing  software  to  local 
needs,  and  special  security- 
sensitive  work? 

SS-423  Conduct  Contracting  Out 
Study 

DSB 

1989 

34 

Which  of  the  following  tasks 
be  done  in-house:  Contractor 
maturity  measurement, 
design/code  reviews,  and 

V&V?  If  so,  was  it  beneficial 
to  do  them  in-house? 

Did  it  require  additional 
resources? 

SS-421  Provide  Efficient  front 

End  Loading 

DSB 

1989 
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Question 

Number 

Question 

Recommendation 

Source 

35 

Would  it  be  beneficial  for 
program  office  personnel  to 
stay  with  the  program  longer? 

Have  program  managers  stay  with 
programs  at  least  through  Beta 
testing  to  maintain  continuity  of 
understanding  of  original  nuances 
of  requirements 

DSB 

1994 

36 

Would  the  rotation  of 
government  and  contractor 
personnel  between  the  PM 
and  the  developing 
organization  be  beneficial  to 
software  development? 

Rotate  government  and  contractor 
personnel  between  the  PM  and 
development  organization  to  build 
understanding  and  trust; 
encourage  use  of  IPA's  fonn 
industry 

DSB 

1994 

37 

How  important  is  early  user 
involvement  in  software 
development  programs  and 
what  is  the  nature  of  the 
relationship  in  your  program? 
How  involved  are  they? 

Require  early  interaction  between 
user,  acquisition,  agent  and 
developer;  identify  and  get  early 
user  involvement 

DSB 

1994 

37 

User  involvement  should  be 
tailored  for  each  program,  varying 
form  cases  requiring  very  limited 
involvement  to  ones  in  which  user 
will  assume  lead  role 

NRC 

1989 

37 

Acquisition  project  managers 
activities  are  communicated  to  all 
stakeholders 

GAO- 

04-722 

37 

Revise  procedures  encouraging 
interaction  between  user  and 
developer  and  achieving  early 
functionality 

DSB 

1994 

38 

How  do  you  know  what 
constitutes  thorough  DT&E? 

The  Air  Force  should  consider 
revision  of  AFR  800-14 
paragraph  5-3,  Test  Planning  , 
and  all  derived  directives,  to 
require  demonstration  of  testing 
of  every  instruction  within  the 
software  prior  to  completion  of 
development,  test,  and  evaluation 
(DT&E).  Implementation  needs, 
costs,  and  expected  benefits 
should  be  analyzed  by  experts 
prior  to  implementing  revisions 

NRC 

1989 

81 


Question 

Number 

Question 

Recommendation 

Source 

39 

Should  software  be  directly  be 
fielded  from  test  beds  if  given 
user  consent? 

Tailor  operational  testing  to 
develop  DoD  "Beta  Test" 
philosophy 

-Allow  fielding  of  software  direct 
form  test  beds  with  user  consent 

DSB 

1994 

40 

Where  future  maintainers 
your  software  product  brought 
in  to  do  V&V  during  software 
development? 

Each  program  should  consider 
using  the  designated  software 
"maintainer"  (operational  phase) 
as  the  independent  validation  and 
verification  agent  during  software 
development 

NRC 

1989 

41 

Who  is/will  be  performing  the 
Operational  Test  and 
Evaluation? 

Have  Facilities  been  provide 
for  the  completion  of  this 
testing? 

Each  service  should  provide  its 
software  Using  Commands  with 
facilities  to  do  comprehensive 
operational  testing  and  life-cycle 
evaluation  of  extensions  and 
changes. 

DSB 

1987 

42 

Do  software  issue  need  to  be 
covered  earlier  in  the 
lifecycle? 

Revise  Milestones  for  Software- 
Intensive  Development 
-Address  the  need  for  software 
first  philosophy 
-Provide  for  a  layered 
software/hardware  standards 
based  architecture 

DSB 

1994 

42 

SS-132  Conduct  Integrated 
Software  Planning 

DSB 

1989 

43 

Who  is  going  to  maintain  your 
software? 

How  do  you  evaluate  the 
efficiency/benefits  of  in-house 
software  support  over 
contractor  software  support? 

SS-424  Measure  Efficiency  of 
Current  LCSE  Centers 

DSB 

1989 

44 

Was  a  plan  developed  for 
software  maintenance? 

SS-313  Provide  for  Management 
of  Software  Change 

DSB 

1989 

44 

Who  is  going  to  maintain  the 
your  software  system? 

SS-422  Consider  Alternative 
Support  Options 

DSB 

1989 
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Question 

Number 

Question 

Recommendation 

Source 

44 

Was  a  plan  developed  for 
software  maintenance? 

Require  planning  for  maintenance 
at  beginning  of  development 
process 

DSB 

1994 

45 

Do  you  have  a  designed 
process  for  release  of  software 
that  is  ready  to  be  fielded, 
block  increments,  or 
improvements?  If  so,  has  it 
helped  reduce  cycle  time  in 
development  and  release  of 
the  software? 

SS-324  Address  Software  as  part 
of  a  Materiel  Release 

DSB 

1989 

45 

Modification  or  upgrades  to 
deployed  versions  of  systems 
components  are  based  on 
deliberate  and  thorough  research, 
analysis,  and  evaluation  of  the 
components  interdependencies 

GAO- 

04-722 

45 

Changes  affecting  how  users  will 
be  expected  to  use  the  system  to 
execute  their  jobs  are  actively 
managed 

GAO- 

04-722 

45 

SS-325  Develop  Responsive 
Distribution  Processes 

DSB 

1989 

46 

Do  you  have  a  formal  process 
to  identify,  track,  and  assign 
problems  in  your  software 
development? 

SS-314  Establish  Internal 

Controls  and  Feedback 

DSB 

1989 

83 


27/30 

What  metrics  are  provided  to 

To  ensure  DoD  has  the 

GAO- 

you  by  the  contractor? 

knowledge  it  needs  to  oversee 

04-393 

a)  How  do  you  use  them? 

software-intensive  acquisitions, 

b)  How  often  are  they  used? 

we  recommend  that  acquires 

c)  Would  it  be  better  to  get 

require  software  contractors  to 

them  more  often? 

collect  and  report  metrics  related 

d)  Would  it  benefit  you  to  get 

to  cost,  schedule,  size, 

metrics  related  to  cost, 

requirements,  tests,  defects,  and 

schedule,  size,  requirements, 

quality  to  program  offices  on  a 

tests,  defects,  and  quality  to 

monthly  basis  and  before  program 

program  offices  on  a  monthly 

milestones  and  that  acquirers 

basis  and  before  program 

should  ensure  contractors  have  an 

milestones? 

earned  value  management  system 
that  reports  cost  and  schedule 
infonnation  at  a  level  of  work  that 
provides  infonnation  specific  to 
software  development 
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Appendix  D.  Interview  Responses 

Question  1 


Yes  through  ASC 

A.  Yes  I  think  so 

11 

I’m  sure  there  is  but  hard  to  say 

Have  not  seen  a  clear  policy  in  a  long  time,  since 
the  acquisition  refonn  nothing  has  come  out  in  a 
while 

A.  No  I  would  say  not 

12 

Yes  there  is 

A.  IT  Lean  supposed  to  fix  the  5000  series 
problem,  ability  to  react  to  small  spirals  in  5000 
is  lacking 

Yes,  tons  of  policy. .  .too  much  policy  and  policy 
often  conflicts 

A.  Sure  plenty  of  guidance 

13 

No,  very  confusing 

A.  No 

Policies  are  more  directed  towards  reporting  and 
manning  ,  EVM  and  Risk  Management  are  in 
ASC  policies.  (Management  of  Software  is 
basically  the  same  as  hardware) 

A.  Sure.... yeah 

14 

Yes-  probably  too  many 

A.  It  radically  changes.  Lots  of  research  or  ask 
the  ACE.  There  probably  is  a  database,  but  not 
easily  accessible.  No  probably  not,  need  to 
consolidate  current  policy  and  make  consistent 
with  each  other. 

Yes 

A.  As  an  ACTD  don’t  adhere  to  all  of  it,  but  yes 
there  is  though 

15 

Yes 

A.  Yes 

I  don't  know,  hard  to  say,  program  is  stable,  there 
is  a  ton  of  policy 

A.  Yes 

16 

Yeah  there  is,  but  not  updated  for  a  while 

A.  Mostly  common  sense 

Yes 

A.  Yes,  but  not  necessarily  agree  with  it. .  .it’s  not 
accurate.  Problem  is  with  estimation.  During 

17 

Yes,  EN  has  some  ,  plus  AF 

A.  Yes  in  general  we  do 

Clearer-  but  still  evolving,  but  that  is  not  a  bad 
thing. .  .need  agile  practices 

A.  Yeah  reasonably  well,  with  the  Perry  refonn 
we  got  ride  of  mil  stds.,  but  pendulum  may  have 
swung  to  far 

18 

No,  it's  getting  there 

A.  No 

Yes 

A.  Yes,  AF  Deskbook  or  contact  EN  Home 
office 

19 

In  written  form,  too  specific  at  times,  others  not 
specific  enough 

A.  Generally  speaking,  some  up-to-date,  but 
constantly  changing 

No  clear  acquisition  policy,  obviously  they  are 
out  there,  but  not  clear 

A.  Don't  use  policy/guidance. . .  .we  use  best 
practices  and  performance  based  specs. 

20 

Yes 

A.  Yes 
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Question  2 


1 

Don't  know  what  it  is,  have  seen  impacts  though 

11 

Not  Familiar  with  it 

2 

Not  sure 

12 

Not  aware  of  it-so  its  like  a  lot  of  the  NDAA  cert 
stuff  or  is  it  talking  about  making  sure  people 
are  qualified;  if  so  yes  but  to  a  lesser  degree 

3 

Yes  and  have  been  good  things,  then  there  are 
things  not  in  804  that  are  driving  things. .  .things 
not  attributed  to  the  act,  but  because  more  people 
understand  software 

13 

Not  aware  of  it. .  .well  yes  and  no. .  .it  is  what  we 
are  doing  right  now,  it’s  what  we  ask  the 
contractor  to  put  in  the  software  development 
plan 

B 

Y eah  in  a  way  (then  referred  to  comments  in 
question  one) 

14 

We  pretty  much  do  that  (she  was  responsible  for 
this  in  her  area,  but  said  most  PMs  would  not  be 
aware  of  it) 

5 

Have  not  seen  impacts,  but  it  looks  good 

15 

Not  aware  of  it 

6 

Don't  know  what  it  is . we  have  documented 

process  for  many  of  the  sections,  may  not  be 
attributed  to  804  though. .  ...just  saw  recent 
guidance  on  software  development 

16 

Not  familiar  with  it,  Software  letter  came  out  of 
Sambur’s  office,  but  it  didn’t  really  say  anything 

new 

1 

Yes,  have  seen  impacts.  We  are  doing  a  better 
job  at  planning  of  programs.  Better  job  at 
collecting  the  right  metrics. 

17 

Seen  it  from  Mr.  Nicols 

8 

Not  aware  of  it 

18 

Have  IT  Lean,  getting  there  on  software  (SAM 
courses  at  DAU) 

9 

Haven’t  really  seen  it  (Section  804),  -  Once 
program  has  started  it’s  too  late,  -  If  starting  a 
program  now,  it  might  be  more  beneficial,  that’s 
when  you  see  where  the  policy  is  at,  at  that  time 

19 

Have  not  heard  of  it,  but  have  seen  impacts, 
makes  it  more  difficult  are  working  level 

10 

Yeah  have  seen  some  impacts. .  .we  do  have  a 
program,  not  sure  if  it  is  specifically  intended  to 
improve  software  acquisition 

20 

N/A 
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Question  3 


Have  not  encountered  any  yet 

A.  Following  guidance  from  EN  Directorate  and 
Center  specific 

11 

Don’t  see  any  areas  -  Contractors  make  their 
own  standards  anyways 

I  would  think  so,  commercial  market  seems  to  be 
more  current  and  has  better  practices 

12 

Multi  level  security  was  killing  us-  commercial 
market  probably  working  on  it 

Not  suggested. .  .would  affirm  that. .  .commercial 
market  drives  everything.  DOD  takes  to  long  to 
develop  things. .  .by  then  they  are  out  dated 

13 

Yeah  we  need  to  stay  abreast  of  most  current 
technology  but  try  to  stay  away  from  military 
standards,  it  may  not  give  the  contractor  to  much 
flexibility 

Commercial  is  hardware  side  of  the  program, 
Unique  parts  of  DOD  is  software,  so  it’s  all 
unique 

14 

Defiantly  agree,  but  it  is  really  tough  to  stay  up 
with  current  technology.  1 101-63X  -Specific 
Training  Programs.  Select  the  leaders  of  the  pack 
or  use  Gardner  Research  (independent  research 
company)  (http://www.gartner.com/).  This  is 
accomplished  in  the  labs  (AFRL) 

No  that  I’m  aware  of 

15 

Yes-absolutely 

Hardware  is  way  behind  the  commercial 
market. .  .software  is  using  a  lot  more 
commercial  tools,  but  we  are  moving  away  from 
contrator  proprietary  development  tools 

16 

Can’t  think  of  areas  that  the  commercial  market 
is  not  covering,  at  least  not  in  aeronautical 

We  use  COTS  and  stay  close  to  industry 
processes  (CMMI,  ISO  9000). .  .we  are  pretty 
close 

17 

True,  almost  mandated  by  leadership,  but  we  are 
getting  away  from  standards  like  Ada 

Small  market  so  will  be  tough  to  do,  More 
(AFRL)  labs  like  Rome,  NY  or  SEI,  Defiantly 
collaboration,  in  past  we  went  our  way  and  now 
we  can’t  support  in  the  commercial  market 
(mentioned  Ada),  -Whether  they  do  what  you  ask 
them  to  do  is  tough,  because  we  are  such  a  small 
market  (1-2%) 

18 

Not  that  I  know  of,  were  looking  at  meta  data, 
and  incorporating  them  ,  so  much  larger  than  the 
commercial  market 

We  are  staying  abreast  of  the  current  technology. 

19 

Mostly  agree,  commercial  market  is  driving 
technology,  but  not  true  for  requinnents  for 
secure  systems.... DoD  drives  this. 

Focus  on  safety  critical,  a  lot  of  our  proccesses 
come  from  the  auto  environment 

20 

No 
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Question  4 


Previous  programs  have  done  it 

11 

Y es  Absolutely,  commercial  market  is  surpassing 
what  the  labs  (AFRL)  and  government  centers 
are  doing.  So  we  (Govt.)  need  stuff  to  leverage 
from  and  “it‘s  about  how  flexible  you  are  in 
changing  software” 

Yes,  some  are  actually  really  good  products,  but 
sometimes  it's  not  all  ways 
ruggedized/militarized 

12 

If  option  is  to  build  it  or  use  commercial 
application. .  .use  civilian  application,  the 
challenge  is  mindset,  if  necessary  have  the 
commercial  market  modify  it. 

Yes,  absolutely.  Tools  today  are  far  superior  to 
anything  before 

13 

Yes,  but  can’t  completely  depend  on  COTS 
products,  want  to  go  that  way  but  no  clear 
guidance 

Yeah  but  the  software  of  the  aircraft  is  very 
unique.  In  transitioning  parts  and  pieces  to  COTS 

14 

Y eah  anytime  we  can  leverage  what  is  existing. 
Have  to  an  understanding  of  the  requirements, 
and  a  clear  understanding  of  the  tool  to  make 
sure  the  tool  and  requirements  match.  Yes  they 
make  suggestions  all  the  time,  but  don’t  drive 
what  the  contractor  uses 

Yes,  based  on  requirements  and  what  you  need  to 
do  the  job 

15 

Y es,  take  advantage  of  their  market 
share . cheaper,  faster,  don't  reinvent 

Yeah,  its  too  expensive  to  maintain  proprietary 
software;  too  unique  and  hard  to  upgrade 

16 

Need  to  rely  on  commercial  tools  that  are 
available  and  build  only  when  necessary.  Rely  on 
contractor  unless  we  direct  them  otherwise 

Always  be  open  to  what  industry  has  to  offer,  but 
be  careful 

17 

Tools,  Methods,  and  Environments  —  Yes. 
However  COTS  is  overblown,  if  you  change  one 
line  of  code  then  lots  more  testing  is  required 

Defiantly  otherwise  support  tools  are  not  there, 
They  become  obsolete  pretty  quickly  so 
constantly  changing,  Will  not  hesitate  to  tell  us 

18 

Yes 

We  should  be  looking  for  these  products  in  some 
areas  to  be  compatible  with  and  to  evaluate  the 
contractors. .  .it  really  depends  who  is  doing  the 
development,  use  the  same  as  the  contractor,  but 
it  is  a  very  small  segment  where  we  are  doing  the 
development. 

19 

Yes,  absolutely.  DoD  is  not  in  the  business  of 
developing  tools,  methods,  etc. 

Yeah,  the  less  we  have  to  develop  from  scratch 
the  better  off  we  are. .  .except  for  safety  critical 

20 

Yes 
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Question  5 


Not  totally  effective  in  best  practices 
dissemination,  it  may  be  there,  but  not  told  where 
to  get  it 

11 

Yes,  but  hard  to  transfer  to  other  programs 

Even  though  they  send  it  out  most  people  really 
don’t  look  at  it 

12 

No 

DOD  has  a  real  desire  to  collect  best  practices, 
the  people  who  try  do  the  best  they  can. .  .have 
been  effective  at  collecting  but  not  disseminating 

13 

Don’t  see  it  happening 

Government  tries,  but  typically  there  is  so  much 
turn  over  of  personnel  we  are  learning  lessons 
over  and  over  again 

14 

They  try  to  disseminate  it,  but  so  much  out  there 
it  is  hard  to  keep  up  with  it.  There  probably  is  a 
tool  out  there,  but  not  aware  of  a  specific  one 
(just  too  much  information  out  there) 

Not  to  sure,  could  be  better 

15 

Effective  in  the  government  perspective,  not 
necessarily  industry  . .  .industry  varies  widely 

Not  as  effective  as  it  could  be,  past  5-6  years  we 
have  been  doing  better  than  in  the  past. . .  .there 
are  plenty  of  people  with  knowledge,  seek  them 
out. . .  .not  sure  if  it  should  be  a  requirement  of 
someone  be  in  charge  of  the  collection. . .  .we 
have  a  best  practice  website,  but  there  are  no 
incentives  to  contribute  to  it 

16 

1  think  we  have  been  good  at  collecting,  but  not 
necessarily  disseminating.  Yes.  None  that  1  am 
aware  of,  but  it’s  up  to  the  contractor  to  use  them 

Starting  to  do  it  smarter  now,  we  used  to  have 
databases  now  we  try  to  build  them  into  the 
program 

17 

ASC  has  and  experienced  people  also  help 

We  do  it  but  not  very  well,  More  R&D  for  more 
agile  acquisition,  AEIT  could  be  used,  studies 
like  this  could  get  them  out 

18 

No...  not  aware  of  tools,  can  use  COPs  but  those 
are  going  away 

Would  say  so,  yes,  AF  Deskbook  is  pretty 
effective  or  Knowledge  Now 

19 

Do  a  fair  job  at  it 

Nah,  its  out  there,  just  have  to  go  out  and  look 
for  it 

20 

Yes 
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Question  6 


N/A 

11 

Don’t  know,  “It’s  a  nice  to  have”,  a  role  for 
maybe  the  ACE,  AFIT  or  Labs 

No  Process 

12 

Can  use  someone  like  MITRE  to  evaluate,  but 
there  is  no  real  process 

If  there  is  I  don’t  know.  What  or  who  would  it 
be?  What  is  the  best  way,  you  can  have  multiple 
perspectives? 

13 

Not  aware  of  one 

We  have  Suites  (releases)  so  we  look  from  Suite 
to  Suite  on  how  to  improve 

14 

Don’t  think  there  is  —  we  look  at  what  industry 
does  and  it  works  for  them  ,  but  not  us  (DoD) 
and  this  can  get  us  in  trouble 

Not  at  my  level 

15 

Yes,  always  look  for  lessons  learned,  but  not 
always  done,  industry  encourages  it  more  than 
government 

Don't  know 

16 

No  process  that  I’m  aware  of. .  .ad  hoc  surveys 
maybe  the  best  thing  though 

Yes 

17 

Didn’t  really  use  metrics,  but  looked  at  processes 

Treated  kind  of  whimsically 

18 

Use  Gartner  processes 

Too  some  extent  the  EN  home  office  is  involved 
in  initial  stages  of  program  or  if  program  is  in 
trouble  a  review  team  may  look  at  things 

19 

Depends  on  who  is  doing  the  software 
development,  younger  developers  may  bring  it 
with  them 

Not  that  I  know  of. .  .just  talk  to  others 

20 

Yes 
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Question  7 


Yes  A.  I  think  so,  it  has  worked  on  other 
programs 

11 

Yes  Definitely 

Yes 

A.  It  would  be  nice 

12 

Absolutely,  gives  you  the  ability  to  pop 
something  out  and  let  them  play  with  it  and  then 
fix  it  in  the  next  spiral. .  .small  spirals  are 
defiantly  the  way  to  go 

SCRUM  might  be  better 

A.  In  some  respect,  in  aircraft  software 
development  it  might  be  the  only  model 

13 

Yes,  especially  with  our  current  funding  issue 

Sure  it  can  be  used,  but  on  a  development 
program,  we  are  more  into  modifications 

14 

Yes  we  should  use  it 

A.  Not  primary  but  should  be  looked  at  more  and 
more,  N/A-  Legacy  system  ,  Usually  driven  by 
data  that  is  available,  Contractor  would  not  help 
decide 

Yes  it  is  appropriate 

A.  Depends,  base  on  speed,  need  and  complexity 

15 

Absolutely,  its  essential  to  be  able  to  do 
simulation  and  prototyping,  to  get  customer  buy 
in 

A.  Don't  know 

Yes,  they  are  so  complex,  you  can't  wait  till  the 
end  to  see  if  it  works,  there  is  no  way  around 
it. .  .have  to  do  this  in  small  increments 

A.  I  guess  so 

16 

Yeah  or  spiral... what  ever  the  buzz  word  is 

Absolutely. .  .risk  reduction 

A.  Depends  on  what  you  are  trying  to  do. .  .early 
in  the  phase,  then  move  to  spiral. 

17 

For  safety  critical  systems  it’s  not  appropriate,  it 
can  put  out  partial  systems,  for  others  systems  it 
may  work 

A.  See  Above  Answer 

Yes,  roll  product  out  in  pieces,  usually  due  to 
funding 

A.  Yes  in  a  way,  mainly  due  to  funding.  -Funding 
use  spiral.  Technological  use  Evolutionanary 

18 

Yes 

A.  Yes  technology  is  changing  too  quickly  can't 
do  big  bang.. to  costly 

Y eah,  it  depends  on  time  frame  and  maybe  not 
on  small  programs,  yeah. 

19 

Yes 

A.  No  depends  on  application 

Yes,  we  use  model  and  sim  extensively 

A.  Sure 

20 

Yes 
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Question  8 


Tailored 

A.  N/A 

B.  N/A 

C.  Don't  think  so 

11 

Should  be  standardized 

A.  Maybe  tailor  for  Requirement  Stability  and  % 
new  development 

B.  It  depends,  but  it  goes  back  to  requirement 
stability  and  %  of  new  development 

Type  of  user  involvement,  but  not  amount 

Things  change  so  fast 

A.  Y eah,  I  guess  you  can 

B.  Yes 

C.  No 

12 

Yes  tailor  programs,  but  requirements  process 
should  remain  standardized 

A.  N/A 

B.  Absolutely 

C.  Looking  to  a  milestone  B  type  thing  (talked 
about  who  makes  program  decisions),  should 
tailor  who  is  make  the  decisions  especially  for 
small  programs 

Tailored  for  each  contractor,  so  yes  for  each 
system 

A.  Yeah  these  are  appropriate  inputs  on  how  to 
tailor 

B.  Yeah 

C.  Policy/Guidance  doesn’t  follow  what  really 
works 

13 

If  there  is  a  standard  process,  do  we  have  enough 
resources  to  do  it,  we  need  clear  guidance  from 
top  level  on  what  to  do  in  software  development 
B.  Yeah  give  some  flexibility 

It  would  be  hard  to  standardize  across  each 

program 

B.  You  have  to  have  them  through  out,  upfront 
so  they  tell  you  what  they  want  and  then  through 
out  development  and  testing 

14 

Yes  Tailor,  each  program  is  not  one  size  fits  all 

A.  Yes  same  as  above 

B.  Yes  to  keep  customer  involved 

C.  Very  limited  ESC  has  a  checklist  for  SE 
processes  that  helps 

Should  be  tailored 

A.  Good  things  to  look  at 

B.  Yes  absolutely  user  involvement  is  good 

C.  No  policy  or  guidance,  it’s  a  grey  area 

15 

Yes,  has  to  be  tailored 

A.  Process  should  be  address  all  methods 

B.  Don't  tailor  by  it,  but  you  defiantly  want  it 

C.  No,  not  aware  of  a  policy  for  tailoring 

Tailored  assumes  you  are  going  off  a  standard  to 
start  with 

A.  These  really  determine  how  you  do  the 
acquistion  and  devlopment 

B.  1  don't  think  in  any  significant  way 

C.  Not  that  I  know  of 

16 

Tailor  for  each  system.  Small  systems  don’t  need 
all  the  artifacts  and  reviews.  Still  need 
requirements  etc.  but  not  all  the  specifications 

A.  Can’t  say  universally  a  way  to  tailor,  there  is 
no  cookie  cutter  approach,  look  at  by  a  case-by- 
case  basis 

Tailored  for  each  system 

A.  Yes 

B.  Have  to  be  involved 

C.  Informal  guidance 

17 

Safety  critical  —  No  there  are  lots  of  things  that 
are  required. .  .waterfall  maybe  the  best  model 

B.  They  need  to  be  involved  to  ensure 
requirements  are  met 

C.  No,  most  things  I  read  tell  you  to  tailor,  but 

A.  N/A 

B.  Technological  use  Evolutionary 

C.  Have  policies,  but  not  always  executed  to 
them 

18 

Yes,  standardized  and  tailored 

A.  Y es  tailor  for  those,  not  sure  about  contract 
and  sport  strategy.  Most  of  what  we  are  looking 
at  is  COTS 

B.  Yes  if  based  on  above 

C.  Not  yet.  Getting  there  quickly 

Should  be  tailored 

A.  Yeah  could  be  tailored  based  on  these 

B.  Yeah  defiantly,  it  can  very  quite  a  bit 

C.  Not  in  depth,  no  real  specific  guidance 

19 

Should  have  flexibility,  do  it  the  best  way  to 
what  you  are  trying  to  accomplish 

A.  Have  to  be 

B.  No  we  don't  involve  the  user  enough 

C.  Don’t  think  so 

Tailor. .  .technology  and  laguages  are  different 

A.  Yeah 

B.  Yeah,  I  would  say  so 

C.  No  guidance  out  there 

20 

Yes  when  possible 

A.  N/A 

B.  Yes 

C.  No 

Question  9 


No  sure  if  it  is  a  polcy,  but  have  used  a  maturity 
model  . . . . ASC  version  of  CMMI 

11 

Don’t  Know. . . .CMMI  model  is  an  indication  but 
the  model  may  show  one  part  a  CMMI  5  ,  but  not 
all  divisions  are  at  the  same  level  so  it  may  be 
misleading 

Yes 

A.  Yes  I  think  so,  in  most  cases 

12 

No  requirement,  something  that  can  be  looked  at 
though 

A.  Have  seen  level  5  program  kick  out  crap, 
while  a  CMMI  level  1  program  kick  out  great 
stuff  -It’s  just  tool  to  help 

People  think  there  is,  but  there  is  none. 
Contractors  think  there  is  one  too 

A.  Great  set  of  recommendations,  but  not  sure  if 
it  is  really  helping 

13 

Heard  about  a  policy,  but  have  not  seen  anything 
written  (mentioned  CMMI  Level  3) 

A.  Doubtful. .  .how  do  they  really  get  to  level  3? 
Don’t  depend  on  it  but  can  be  useful 

Yeah  the  basic  standards. .  ..occurs  in  the 
proposal 

14 

Policy  for  dealing  with  companies  not  for 
government.  Just  cause  contractor  can  do  it 
(process)  doesn’t  mean  its  repeatable 

A.  It’s  debatable. 

Yes  there  is  a  policy,  with  TRLs  there  is  a  right 
time  to  go  to  production 

A.  Yes 

15 

Y eah,  you  set  requirements  for  it  before  you 
proceed  into  milestones 

A.  No,  we  do  what’s  smart... if  not  mature  enough 
you  don’t  go  forward 

Has  been  proposed  before,  we  at  ASC  are  not 
propenents  of  maturity  levels,  they  do  not 
gurentee  good  products 

A.  there  are  other  things  at  ASC  that  are  more 
effective 

16 

I  believe  there  is,  most  are  CMMI  level  3 

A.  Not  necessarily,  they  will  be  at  a  level  5  but 
not  following  the  process  in  place 

CMMI  level  3 . .  .unofficially,  but  if  an 
organization  is  big,  all  parts  of  organization 
might  not  be  at  the  same  level 

A.  Yes,  it  reduces  the  lighting  for  information 

17 

I  think,  we  do. ...level  3  CMMI. . ..can’t 
remember  AFI 

A.  Can’t  remember  any  program  that  comes  near 
cost  and  schedule 

Yes 

A.  Just  cause  one  part  of  the  company  is  highly 
certified  another  part  of  the  company  may  not  be 

18 

There  is  but  people  are  not  following  it  they  don't 
understand  TRLs 

A.  If  used  yes,  but  not  being  used 

More  organizations  need  to  have  a  maturity  level, 
but  have  not  looked  at  guidance  recently,  but  a 
few  years  ago  there  was  some  debate 

A.  No  visibility  into  this,  but  it  gives  a  false 
sense  of  security,  because  in  a  big  company  one 
division  may  have  reached  a  certain  level,  but 
others  division  that  you  are  working  directly  with 
may  not  have 

19 

Yes  I  think  so,  depends  on  the  application,  if 
doing  business  applications  yes,  imbedded 
systems  are  a  little  tougher 

It  is  desired,  but  not  nessecarily  required. . . .its  a 
factor  maybe  a  policy  out  there  that  says  have  to 
CMMI  level  3 

A.  I  don't  know,  certanly  hasen't  helped  ours 

20 

Did  not  apply  to  their  program 
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Question  10 


Yes  on  technical 

11 

Absolutely,  we  focus  on  the  schedule  and  cost 
that  are  in  a  bid 

Yeah  it  would  be  nice  to  do  technical,  but 
technical  knowledge  of  the  government  is  not 
always  there  to  do  this 

12 

Depends  on  when  you  want  to  go  with  it,  from  a 
life  cycle  perspective  it  is  feasible 

I  think  it  is,  but  hard  to  do 

13 

Should  be  a  balance  of  all  of  them 

We  are  IDIQ  (Indefinite  Delivery/Indefinite 
Quantity)  so  same  contractor  for  the  next  15 
years  or  more 

14 

We  focused  on  technical  approach;  it  had  a 
heavier  rating  in  source  selection 

Y  eah  depends  on  what  you  are  working  on 
(requirements,  missions) 

15 

No  it's  a  trade  off 

Yeah,  we  have  do  that  in  source  selections,  it’s 
the  best  value  source  selection  theory. .  .most 
bang  for  your  buck 

16 

It  is  feasible,  and  should  be  done,  but  probably 
won’t 

Yeah,  I  would  rather  overbid  than  underbid  and 

overrun 

17 

Yes,  may  not  play  a  role  in  who  is  selected. .  .lots 
of  political  influence 

Yes,  classified  programs  often  look  at  technical 
merit  rather  than  cost 

18 

Yes,  it's  the  best  value  approach 

Defiantly,  technical  approach  is  at  least  as 
important  as  cost  in  many  projects 

19 

Yes  you  have  to  do  it,  have  been  on  a  source 
selection  where  too  much  focus  was  on  cost,  not 
enough  technical 

Yes,  better  technical  approach  it  maybe  costly 
upfront  but  save  in  the  end 

20 

Did  not  apply  to  their  program 
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Question  1 1 


No  expensive 

11 

Yeah  good  idea 

Y eah  I  would  like  to  see  it 

12 

Yes,  it’s  market  research 

Hard  to  due  since  major  requirements  might  not 
be  there 

13 

Yes,  if  you  have  time  to  do  independent  market 
analysis 

Y eah  if  it  is  not  core  software 

14 

Yes 

A.  The  team  worked  with  an  independent 
contractor  to  walk  through  the  process  of 
determining  best  value 

Yes 

15 

It  could  be  beneficial  if  there  is  something 
already  out  there  and  meets  the  requirement  we 
should  use  it 

Yes  before  you  write  any  RFP,  you  find  out  what 
is  avilble  our  can  actually  be  done 

16 

It  is  feasible,  and  should  be  done,  but  probably 
won’t 

Yes  Absolutely 

17 

Yes,  but  with  flight  controls  hard  to  reuse 

Yes  we  have  been  doing  market  analysis,  Look  at 
ours  for  similar  programs  and  some  gut  feeling 
because  no  two  programs  are  identical 

18 

Good  to  do,  should  do  it  any  how 

Could  be  beneficial,  but  we  should  have  policy 
on  what  does. . .  Yes,  Quantitative  trade  off 
analysis 

19 

N/A 

It  dosen't  hurt  to  do  it 

20 

Did  not  apply  to  their  program 
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Question  12 


Yes  it  was  helpful,  contractor  had  a  prototype 

A.  It  was  not  considered,  not  a  driver 

11 

Y  eah  they  ought  to  be  able  too 

A.  But  often  there  are  new  things  that  might  not 
be  out  there,  so  it’s  completely  new  development 

It  would  be  nice,  things  done  in  the  past  really 
show  what  they  can  do 

A.  It's  been  good 

12 

It’s  a  whole  different  acquisition  strategy,  may 
not  be  the  best. .  ..well  it  depends 

A.  You  look  at  technical  approach  and  how  they 
are  going  to  use  it  -  Some  use  COTS  and  then 
build  their  own  interface 

I  think  so,  good  idea 

A.  Have  not  been  in  a  source  selection 

13 

Yes 

A.  Yes  it  was  considered 

N/A 

14 

We  are  always  looking  for  COTS  solutions  and 
will  tailor  around  that 

Yes  it  would  be  beneficial 

A.  The  weight  was  not  as  high  as  other  aspects 

15 

Yes,  otherwise  you  don’t  know  the  level  of 
maturity  otherwise 

A.  Should  be  considered,  past  performance  in 
using  COTS  ,  Need  to  know  the  pitfalls  of  COTS 

Heck  Y eah,  more  reuse  the  better 

A.  N/A 

16 

Doesn’t  matter 

A.  Still  need  to  do  requirements  and  integration, 
so  not  really  saving  anything 

Yeah 

A.  Make  sure  it’s  COTS  or  Reuse,  but  if  they 
make  major  changes  I  don’t  consider  it  COTS  or 
Reuse. .  .has  to  be  truly  COTS  to  be  beneficial 

17 

For  ground  based  stuff  yeah 

Definitely 

A.  See  a  lot  more  of  it  now 

18 

Yes,  did  it  a  lot  in  the  90s,  get  technical  value 
added 

Might  go  to  far.. .in  general  re-use  can  be  over 
optimistic  on  how  much  is  going  to  be  reused 

A.  Look  at  developing  organization  if  proposing 
COTS 

19 

Yes,  if  they  can  they  should  demo 

Y  eah,  w  are  in  the  demonstration  world  today, 
we  like  to  see  things  work 

A.  Considered  a  lot  in  cost,  schedule,  time  and 
level  of  effort 

20 

Did  not  apply  to  their  program 
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Question  13 


No 

11 

No 

No 

12 

No,  Engineer  and  Market  Research,  Gartner 
Research  is  often  used 

Never  seen  one,  might  exist 

13 

Don't  have  one 

We  only  have  once  piece  of  COTS  on  the  whole 
aircraft 

14 

Do  fly  offs  were  competitors  come  in  a 
demonstrate  also  went  out  to  industry  to  seek 
solutions 

Contractor  determines  based  on  requirements 

15 

Lists  are  available  with  vendors 

No 

16 

Not  really. .  ..just  Google  search  or  other  ad-hoc 
searches 

No  we  don’t 

17 

Had  a  database  a  while  ago,  or  at  NATCOM 
(conferences),  now  we  rely  on  contractors  to  do 
this  more 

Talk  to  others  in  the  organization  and 
industry. .  .basically  word  of  mouth 

18 

Through  market  research,  small  businesses,  put 
word  out,  and  Industry  days 

Though  knowledge  of  what’s  out  there  and  web 
searches 

19 

Not  directly,  since  embedded  systems,  other 
areas  no  system  but  the  internet 

We  did  early  on,  don't  so  much  any  more 

20 

Yes 
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Question  14 


Don't  think  so,  it  depends  on  the  application 

11 

Kind  of  agree. .  .puts  risk  on  contractor,  they  are 
maybe  responsible  for  a  part  (of  the  system)  that 
they  might  not  have  created,  and  they  now  have 
to  maintain  it.  Depends  on  level  of  maturity 

No 

12 

Yes,  push  back  to  ensure  do  diligence  is 
incorporated  into  acquisition  strategy 

In  this  day  and  age  might  be  a  good  idea  in  the 
past  opposite 

13 

The  reality  is  in  the  (program  masked)  we 

assume  so 

No,  if  you  are  buying  a  desktop  PC  sure,  but  not 
for  pieces  for  the  aircraft 

14 

Almost  have  to. .  .to  communicate  beyond  small 
programs  and  to  get  them  AF  wide.  Also  it  will 
help  alleviate  stove  piping 

No 

15 

Where  COTS  is  the  potential  should  always  look 
at  them. .  .If  a  support  trail  is  established  it  will  be 
cheaper 

Very  dangerous  assumption. .  .almost  everything 
we  deal  with  we  modify. . .  .the  software  is  not 
usually  developed  for  the  hardware  we  use 

16 

No,  if  all  COTS,  most  don’t  schedule  enough 
time  and  budget  then  if  you  have  to  modify  you 
end  up  in  trouble 

No,  but  consider  them 

17 

Should  not  assume,  only  through  analysis 

Contentions  due  to  money  issues,  Dual-use  -  if 
COTS  are  available  and  are  qualified  to  use  them 

18 

Not  a  good  assumption,  have  to  do  some  analysis 
to  know  where  you  stand 

PM  should  assume  there  is  going  to  be  unique 
improvements,  Anytime  it  can  meet  the 
requirements 

19 

No,  should  assume  can't  do  it  with  COTS 

I  would  say  no,  I  would  not  make  that 
assumption. .  .have  to  look  at  safety 
criticallity . . .  .have  a  unique  system 

20 

Yes 
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Question  15 


No  real  experience  with  this,  but  it  should  be  the 
last  resort  to  modify  commercial  compenents 

11 

Depends...  were  COTS  products  developed  with 
a  robust  set  of  documentation  and  mastery  of 
development 

No,  shouldn’t  be  discouraged 

12 

Ideally  don’t  want  to  modify,  but  may  have  to 
modify  interfaces,  can  be  costly  and  hard  when 
new  (COTS)  software  updates  are  pushed  out 

Would  not  want  to  discourage  modification.  In 
the  past  told  not  to  modify,  but  today  not  so 
much 

13 

Yes  I  think  so 

When  you  do  a  market  survey,  you  do  some,  but 
this  might  be  too  much 

14 

Should  be  encouraged  more  than  it  is  now 

Yes  for  legal  issues  in  modifying  COTS,  better  to 
build  from  scratch 

15 

Yes,  difficult  once  modified,  you  lose  the  support 
trail  (because  it  is  now  a  unique  system),  You 
can  always  tailor  it  to  be  backwards  compatible 

Not  enough  of  this,  usually  only  looking  t  short 
term  savings 

16 

Anything  you  do  should  be  justified 

It  should  be  discouraged,  because  when  you 
modify. . .  you  are  now  a  development  program. 

17 

Yes,  modification  requires  a  lot  more  testing. .  ..if 
you  modify  it  is  it  still  COTS? 

Defiantly,  if  we  don’t  have  to  change  them  then 
don’t 

18 

Yes,  some  cost  analysis  to  see  if  cheaper  to 
change  COTS  or  business  processes 

Yes  discourage  in  general 

19 

Defiantly  yes,  also  need  a  demo  to  prove  it 

No  shouldn't  be  discouraged 

20 

No 
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Question  16 


Yes  it  is  beneficial,  happened  on  the  previous 
program. 

A.  Yes 

11 

You  have  to  make  trade-offs  can  not  constrain  all 

three 

A.  Yes 

It’s  a  common  practice 

A.  Yes 

12 

Can’t  realistically  constrain  2  of  the  3,  A.  No,  all 

3  have  to  be  fulfilled,  but  there  is  some  trade 
space 

Don't  know  if  we  can  constrain  two  of  three. 

Price  always  constrained 

A.  That  is  what  we  do 

13 

This  happened  price  and  schedule  were 
constrained  . .  .functionality  not  so  much,  A.  It  is 
a  realistic  approach 

Depends  on  the  program  goals 

A.  Nothing  wrong  with  this  though,  could 
constrain  only  one 

14 

We  don’t  constrain  all  of  them 

A.  Reality  is  that  you  get  beat  up  if  you  don’t 
meet  all  of  them,  Depends  on  the  political 
environment 

Need  all  three,  all  the  time 

15 

Have  to  manage  all  three,  A.  Not  realistic 

Everything  tends  to  be  connected 

A.  Usually  not,  constrained  on  all  three 

16 

Don't  really  know  if  it  is  beneficial 

A.  Yes  probably  realistic  but  not  today 

Program  dependant,  depends  on  understanding  of 
requirements  and/or  the  contractor  being  used 

17 

Yes  something  has  to  float  supposed  to  use 

CAIV 

A.  I  don't  like  constraining  schedule 

Yes 

A.  Tough  on  PM,  their  career  can  depend  on  it 

18 

Not  sure...  usually  constrained  with  all  three 

N/A 

19 

Schedule  and  Price/funding  hard  to  manage 

A.  Its  real  life,  not  realistic  to  manage  all  three 

You  have  to  balance  all  three  can't  constrain  only 
two 

20 

No,  not  realistic,  price  is  usually  constrained. 
Schedule  and  functionality  impact  each  other. 
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Question  17 


Have  a  dialogue  with  contractor  and  review  the 
proposal 

A.  N/A 

B.  Do  a  trade  off  analysis 

11 

Through  validation  and  systems  engineering 

A.  Validating  requirement  and  understanding 
trade-offs  of  Price,  Schedule  and  Functionality 

You  never  really  know.  Y ou  go  over  and  over 
them  again  and  again 

A.  N/A 

B.  Yes 

12 

Working  expectations,  also  system  requirements 
reviews,  functional  requirement  reviews,  etc.  A. 
All  of  the  above  plus  user  involvement,  B.  When 
hit  a  critical  junction  (PDR,  CDR)  then  go 
through  them  with  user 

No  one  knows  at  beginning  if  they  are  feasible. 
But  after  time  what  is  infeasible  becomes 

obvious 

A.  By  look  and  feel.  Each  side  will  look  at  them 
in  different  ways 

13 

Very  difficult,  done  through  peer  review 
(program  office,  contractor  and  user),  Peer 
review  and  face-to-face  meetings,  A.  Peer  review 
B.  Yes  done  at  system  level,  maybe  not  all  cases 
(software  level) 

Go  through  requirements  with  contractor, 
program  office  and  user 

A.  Same 

B.  N/A 

14 

Through  IPT’s,  Developed  IPT  to  discuss  these 
items,  No  it  should  be  unique  to  the  program 

A.  Through  the  IPT 

B.  Yes  always,  we  are  pretty  good  at  doing  all, 
that 

Design  reuse  and  some  prototyping 

A.  Software  design  documents 

B.  Typically  yes,  absolutely  should  be  done 

15 

Systems  engineering  approach  and  evaluate,  A. 
Documentation-  all  software  design  documents 
and  requirements  must  be  frozen  Also,  design 
review,  PDR,  CDR,  etc.  and  always  invite  user, 

B.  Analyze  cost  and  schedule  impacts,  you  don’t 
want  to  change  requirements,  but  if  you  do  you 
must  do  an  RFP  along  with  a  Systems 

Engineering  analysis 

Contractor  does  analysis  and  determines  if  it  will 
work 

A.  Meetings  with  contractor  and  pilots,  have 
sims 

B.  Haven't  had  any  major  changes 

16 

Experience,  Engineering  judgment  and  do 
analysis 

A.  You  never  do,  but  can  sit  down  with  user, 
developer  and  program  office 

B.  Sometimes  a  trade-off  is  accomplished  and 

Yes,  have  seen  impacts.  We  are  doing  a  better 
job  at  planning  of  programs.  Better  jobs  a 
collecting  the  right  metrics 

A.  Bring  in  the  user  and  do  requirements 
reviews;  contractors  can  do  internal  peer  reviews 

B.  Yes  already  do  those _ do  them  upfront  for 

risk  reduction 

17 

Testable,  traceable  —all  the  ilities 

A.  Through  the  review  process  contractor  has 
analysis  tools 

B.  Not  done  in  general 

Contractor  will  tell  us,  well  it  depends  on  the 
contactor. 

B.  Not  always  at  Software  level,  usually  at 
systems  level 

18 

Functional  review  boards  that  take  functional 
requirements  to  systems  requirements 

A-  Industry  days  and  build  demo's  with  them 
B-No 

By  knowing  the  state  of  the  art  and  what  can  be 
done  with  commercial  products 

A.  In-depth  discussion  and  in-depth  document 
review 

B.  Yes  there  is  a  process 

19 

N/A 

A.  When  they  deliver  the  product,  in 
development  process  and  demos  should  show 
you  are  talking  about  the  same  thing 

B.  Yes  do  it  constantly 

Don't  always  know  until  implement  and  test  them 
out. . .  .through  discussion 

A.  Through  discussion 

B.  Always,  eventually  software  becomes  the  fix 
because  hardware  is  too  expensive 

20 

A.  Weekly  IPT  meetings,  mature  software 
processes 

B.  Yes 

Question  18 


Acquistion  Community  and  User 

11 

PM 

Customer 

12 

PM  in  conjunction  with  the  User,  a-  Needs  to  be 
at  appropriate  level  (PM  or  PEO  or  MDA),  b- 
Depends  on  program  size  -  really  should  be  at 
appropriate  MDA 

User 

13 

Program  Manager  and  User,  yes 

PM  and  User.  PM  tells  user  what  they  can  do 
and  user  makes  decision 

14 

Customer  and  PM,  Yes  defiantly,  small  programs 
have  more  flexibility,  To  make  an  exception  for 
one  you  have  to  make  an  exception  for  all 
(requirements)  A  PM  can  justify  anything 

Program  office  and  User 

15 

User  and  PM 

Program  Manager  has  authority,  mostly  because 
they  know  the  impacts 

16 

PM  once  requirements  are  established 

If  they  are  customer  requirements  then  the 
development  office  with  input  from  customer 

17 

User 

User 

A.  Depends  more  on  the  environment,  who  the 
product  is  for,  how  import  is  the  system 

B.  PM  often  decides,  then  goes  to  the  user  and 
asks  for  forgiveness 

IB 

User  with  inputs  from  acquisition  community 

PM  with  the  sponsor,  should  go  to  who  ever  has 
authority  and  who  has  review  purposes, 
shouldn’t  matter  if  higher  authority  is 
comfortable 

19 

Only  the  user 

Program  director 

20 

ACC/USER 
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Question  19 


Havent  really  done  anything  yet.  Could  in  the 
future  with  net-centricity 

11 

Don't  really  get  involved 

Yes  it’s  an  excellent  tool 

A.  Yes  it  helps 

12 

Has  its  own  challenges,  but  not  necessarily 
improved  it 

Integrate  into  the  program 

A.  Not  really  improved,  it  is  what  it  is 

13 

No  answer 

Yes  we  have  one  and  when  we  make  a  change  to 
one  piece  we  use  to  look  at  what  else  is  effective 
A.  N/A 

14 

Critical  Role,  but  if  others  can’t  us  it  you  don’t 
gain  anything 

A.  In  the  long  run  it  would  have  to 

Have  poor  software  architecture,  working  on 
implementing  an  architecture 

A.  Should  make  it  cheaper  and  help  with  growth 

15 

It’s  a  huge  role,  directly  determines  complexity 
of  software  design,  gives  you  lots  of  choices  on 
how  you  set  up  your  software  A.  Yes,  always 
look  at  and  re-evaluate  to  make  sure  it  sill  fits 

N/A 

16 

They  are  usually  at  the  system  level 

A.  Don't  know 

They  are  very  important. .  .gives  you  insight  into 
requirements  and  sustainment  feasibility 

A.  Minimal. .  .if  there  is  a  risk  of  breaking,  then 
we  make  a  change 

17 

Absolutely...  mapping  software  to  hardware 
keeps  stuff  manageable 

Commercial  market  has  been  using  them  for  a 
while,  we  are  starting  to  use  them 

A.  Improved  integrity,  but  some  times  its  a 
hindrance 

18 

N/A 

N/A 

19 

Continuous  problem  in  last  10  years  more 
emphasis  not  universally  applied 

A.  It  can  improve 

Built  system  first  then  did  architecture...  did  it  in 

reverse 

A.  It  is  extermely  difficult  to  now  draw  an  exact 
usefull  architecture. . .  It  hasen't  improved 
development 

20 

Constantly  analyze 

A.  Yes 
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Question  20 


No  incentives 

11 

Don't  really  build  a  lot  of  SW ...  So  no 

Just  on  this  program 

A.  N/A 

12 

They  are  a  good  thing  and  should  be  considered 
in  Acquisition.  Strategies. .  .need  to  ask  what 
behaviors  you  are  trying  to  incentivize?  A.  In 
some  cases  yes  incentivize  these,  but  in  some 
case  like  legacy  systems  incentivize  other 
things...  like  up  time  (availability) 

Yes 

A.  It  is  hard  to  measure 

13 

It’s  better  to  have  it,  but  takes  to  much  time  to 
collect  feedback  and  to  many  resources  needed  to 
give  out  fee. .  .in  reality  bypass  it 

Award  fee,  but  we  are  looking  at  incentive  fee 

A.  Don't  really  care  about  these.  Want  to 
incentivise  schedule  and  getting  capabilities 
within  that  schedule 

14 

Award  Fee  -  use  Finn  Fixed  Price 

A.  Tough  to  measure  any  of  these,  Program  was 
cancelled,  Brought  in  someone  from  ACE  to  help 

No  specific  incentives  for  software 

15 

Not  for  software,  but  for  total  program. .  .there 
should  be  incentives  for  software  though  A.  Yes 
for  quality,  but  it  depends  on  what  your  building 

Cost  plus  incentive  fee,  incentive  to  spend  less 
money,  award  fee  used  in  past-  worked  well 

A.  Yes,  if  you  can  make  the  program.  In  the  past 
it  was  beneficial 

16 

Not  really  sure,  some  maybe  on  maintenance  side 
A.  Yes 

Award  fee  A.  During  source  selection  yeah 

17 

They  get  awards,  but  nothing  specific  to  software 
A.  Yes 

Yes,  sometimes  but  amount  not  much  and  can  be 
a  morale  booster 

A.  Hard  to  quantify 

18 

Not  that  I  know  of. .  ..good  idea  though 

Not  really  used  at  software  level 

A.  For  quality  it  makes  sense,  but  for  reuse  it  can 
be  a  trap  if  program  needs  lots  of  changes 

19 

Yes  award  fee 

A.  Yes 

Yes  have  a  specific  award  fee 

A.  Yes  covers  all  of  it 

20 

Yes 

A.  Yes  depends  on  the  software  fix 
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Question  2 1 


1 

N/A 

11 

Yes  not  common  across  all ,  beneficial  to  have 
different  ones  to  costly  to  create  a  standard. 

2 

No,  haven't  used  one.  FM  says  they  have  one 

12 

Yes  there  is. .  .ACID  used  by  ESC 

3 

Have  a  couple,  COCOMO  and  SEER/SEM 

13 

C-SAM-  used  by  engineers  and  Price-S  used  by 
FM,  Better  not  to  have  it,  use  two  to  check  each 
other 

1 

Been  with  same  contractor  for  so  long  it  is  hard 
to  change 

14 

Yes,  done  in  the  FM  community,  Would  be  very 
useful,  but  (models)  are  not  always  accurate, 
good  for  estimating,  but  will  need  to  do  some 
additional  analysis 

5 

No 

15 

Yes,  SCCM  and  Price-S,  Yes  there  should  be, 
find  one  that  works  well,  Don’t  know  them  well 
enough  to  say  which  one  though 

6 

Contractor  has  this  own.  We  have  our  own, 
calibrated  to  past  performance 

16 

Yes  we  use  ?  Pro-  everyone  works  with  same 
model  Con-  if  flaw  everyone  has  flaw 

1 

Yes  SEER/SEM 

17 

Prices- S 

A.  I  think  so  plus  use  independent  people 

8 

Lost  art,  rely  on  several  models 

A.  No  standard  in  industry,  but  should  use  what 
they  use 

18 

Yes  there  is 

9 

Price-S  and  SEER  were  both  used,  You  don’t 
want  one  standard  model  need  to  compare  with 
two  or  three  models 

19 

NG  and  CRC  as  their  standard.  It  would  be 
beneficial  if  we  could  get  everyone  to  agree 

10 

Yes  we  have  a  standard  model 

20 

No,  historical  data 
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Question  22 


1 

No,  we  don’t  track  software  costs 

11 

No 

2 

Someone  in  FM  may  do  that 

12 

Yes 

3 

Yes  we  do 

13 

Don’t  track  software  costs,  we  track  cost  for  all 
acquisition  (cost  by  WBS) 

B 

Yes  we  do  but  life  cycle  restarts  when  you  field  it 

14 

Did  not  get  that  far-  program  cancelled 

5 

No,  things  get  pushed  or  taken  away.  Very 
dynamic  program  hard  to  track 

15 

Yeah,  software  metrics  tracks  Estimate  to 
complete  until  fielded 

6 

No 

16 

Don't  think  so...  track  through  SDD  to 
deployment 

1 

Through  out  development  and  then  through 
sustainment  and  by  two  different  groups 

17 

Should  have  a  database  to  do  this 

8 

Not  very  well,  should  probably  do  it. .  .but  it  is 
hard  to  keep  track  of 

18 

Not  that  I  know  of 

9 

Yes  they  are  defiantly  tracked,  Maybe  good  to 
get  an  independent  look  at  risks. .  .otherwise  new 
risks  may  not  get  identified 

19 

Yes  we  do 

10 

We  do  and  review  it  at  lower  levels  monthly, 
higher  levels  quarterly 

20 

Yes 
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Question  23 


Yes 

A.  Follow  ASC  policy 

B.  Annual/S  emi -annual 

11 

Yes,  used  one  from  contractor  and  tailored 

A.  Yeah  EN  had  guidance 

B.  Quarterly,  but  more  often  when  big  ones 
jumped  up 

Yes 

A.  Yes 

B.  Design  review  or  PMR 

12 

Yes,  A.  Yeah  infonnal  policy,  B.  Not  enough,  at 
a  minimum  should  be  looked  at  major  events 
(milestones)  and  at  monthly  PMRs 

Yes 

A.  Yes 

B.  Quarterly  high  level,  biweekly  lower  level 

13 

Yes,  it  was  built  in  to  system  engineering  plan 
also  we  have  a  SPO  plan  and  a  contractor  plan,  a- 
Don’t  know  about  it,  b-  Monthly-  Risk  Working 
Group  (SPO  and  Contractor) 

Yes  jointly  with  contractor 

A.  Sure  there  is 

B.  Monthly 

14 

Yes  had  one 

A.  Yes  there  was,  but  can  not  quote  exact  policy 

B.  Briefed  at  monthly  PMR  and  discussed  often 
in  the  Program  office 

Yes 

A.  ASC  policy 

B.  Monthly 

15 

Yes,  it  is  essential  to  have,  A.  Required  to  have 
them,  but  don’t  know  if  there  is  policy  on  how  to 
create  it  Always  had  one,  just  change  it  to  make 
improvements  B.  At  least  monthly,  at  higher 
levels  quarterly,  and  lowest  levels  weekly 

We  do  them,  but  may  not  updated  often 

A.  We  have  some 

B.  Weekly  risk  meetings-  day  to  day  risks...  then 
at  quarterly  PMRs 

16 

Yes 

A.  Yes  there  is  a  policy 

B.  Monthly 

Yes 

A.  Yes  EN  provides  guidance 

17 

Yes,  don't  think  there  is 

Yes 

A.  Yes  tracked  under  systems  engineering. .  .PM 
made  it  mandatory 

B.  Quarterly  and  at  bi-monthly  PMRs 

18 

A.  Yes  DoD  policy 

B.  At  least  at  PMR  quarterly,  at  lower  level 
Monthly 

Yes  we  have  one 

A.  Yeah  there  was  standard  policy 

B.  Every  other  week 

19 

Yes 

A.  Yes 

B.  Almost  weekly,  quarterly  comprehensive 
review 

Yes 

A.  Yes 

B.  Lower  Monthly  sometimes  more 
often. .  .higher  quarterly 

20 

Yes,  but  nothing  documented 
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Question  24 


1 

Few  deliverables 

11 

Part  of  the  original  proposals 

2 

Decided  early  in  the  program,  kind  of  a  set 
standard  and  cross  off  what  they  don’t  want  and 
also  work  with  the  contractor 

12 

No  answer 

3 

We  don’t  decide 

13 

Guidelines  from  ASC/EN  posted  on  EN  website 
which  gives  required  documents  by  milestones 

B 

EN  community  decideds  what  they  want  unless 
too  costly 

14 

Never  got  that  far 

5 

What's  in  it?, What  bugs  are  there?, 

Requirements,  Specs,  software  descriptions 

15 

Base  it  on  what  you  need  to  get  approval  of  a 
document,  use  the  sparingly  CDRLs  are 
expensive 

6 

Been  around  for  so  long,  have  a  standard  list. 

Add  new  one  every  now  and  then 

16 

What  has  been  delivered  before,  there  is  a 
standard  set,  most  people  get  same  stuff 

1 

What  ever  gives  you  insight  into  what’s  being 
developed 

17 

Need  proper  documentation  if  going  to 
recomplete 

8 

N/A 

18 

Engineers  decide,  some  guidance  with  SEP  and 

IT  Lean 

9 

There  are  some  things  you  have  to  have  others 
depend  on  the  size  of  the  program 

19 

Did  you  ask  for  enough  or  too  much. .  ..it  depends 
on  what  you  are  doing  and  how  much  risk  is 
involved . if  risky  then  more 

10 

Don't  have  any  on  program... all  performance 
based  specs  and  contractor  sets  them 

20 

Based  on  historical  need 
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Question  25 


No 

11 

No 

A.  N/A 

No,  havent  seen  it  done  in  a  long  time 

12 

Does  not  apply  here 

Yes 

A.  Don’t  know. 

13 

Haven’t  had  one  for  quite  awhile  have  an  action 
to  make  it  come  back, 

No  offical  group,  but  people  keep  an  eye  on  it 

A.  N/A 

14 

Don’t  know 

No 

15 

It  has  one,  but  neglected  for  a  while,  now  being 
reestablished  A.  If  doing  their  job  there  will  be 
no  unexpected  surprises  and  through  the 
readiness  of  the  labs 

No,  we  are  stable  in  what  we  do  and  how  we  do 
it 

16 

No  trying  to  re-established  one 

Government  lead?.... No 

17 

I  think  CRWGs  are  dead,  most  have  IPTs 

Not  anymore 

18 

Not  sure,  may  have  something 

We  had  a  software  working  group 

A.  Not  really  evaluated 

19 

Have  something  similar 

A.  Self-evaluation 

We  don't  have  CRWG  we  have  block  managers 
that  do  something  similar 

20 

Yes 

A.  Based  on  end  product 

Question  26 


Have  not  had  an  IER 

11 

No  not  specific  but  the  contractor  has 

Nt  on  this  project,  still  exists,  EN  home  office 
does  this  (tiger  teams) 

12 

An  independent  V&V  contractor  was  used  on 
other  programs 

Yes,  a  couple 

13 

Yes,  it  depends. .  .when  a  problem  occurs 

They  are  pulling  together  a  team  to  look  at  the 
program 

14 

Didn’t  get  that  far,  did  use  a  SEP  (Systems 
engineering  process)  which  will  track  a 
requirement  through  entire  process 

Yes 

15 

No  well  maybe,  we  have  Independent  Review 
Teams  (IRT)  and  Executive  Review  Team  (ERT) 

Sure,  had  2  or  3... had  an  unexecutable  program  at 
one  time 

16 

No  we  have  not  on  software 

Yes  had  several. .  ..good  to  get  a  different  set  of 
eyes  on  the  program 

17 

Have  safety  review  for  first  flight  and 
independent  teams 

Only  when  a  problem  arises  or  GAO/UCI  visits 

18 

Not  that  I  know  of 

Yes,  we  did  have  an  independent  review 

19 

No 

Yes 

20 

Yes 
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Question  27 


No  metrics 

11 

Cost,  man  hours 

A.  Track  obligation  and  expenditures 

B.  Every  couple  of  months 

C.  Monthly  so  no 

D.  Depends  on  size  of  program  small  programs 
would  be  burden 

A.  Information  only 

B.  Monthly 

C.  Sometimes  thet  can  really  help  others,  if  not 
its  just  good  info,  but  not  really  useful 

D.  We  get  these  and  its  helpful 

12 

Test,  funding,  manning.  Earned  Value,  Help 
desk,  deficiency,  B.  Sometimes  daily,  depends  on 
what  you  are  doing  -if  in  a  test. .  .test  metrics  used 
daily  others  like  Earned  Value  maybe  monthly 

C.  Depends  on  the  metric  D.  Depends  on  the 
metric 

Worm  charts  based  on  IMS 

A.  Plan  vs  actual 

B.  Review  once  a  month 

C.  Yes  but  costs  more  in  lost  productivity  and 
not  feasible 

D.  Already  get  them 

13 

Guidelines  about  metrics  in  2006  letter  (EN 
letter)  B.  Frequently. .  .when  we  have  time  too. 

Bit  they  are  updated  every  two  weeks  C. 

Probably  not. .  .too  costly.  Every  two  weeks  is 
good  enough 

SLOC,  testing,  problem  reports,  EVM,  IMS 

A.  N/A 

B.  Monthly 

C.  N/A 

D.  Get  all  these 

14 

Some  schedule,  didn’t  really  get  any  typical  cost, 
schedule  performance  metrics,  Didn’t  really  use 
metrics,  program  was  well  established  and  only 
focused  on  integration  of  established  programs 

Cost,  schedule,  nothing  software  specific 

A.  EVM 

B.  Program  quarterly,  lower  biweekly 

C.  No 

D.  Yes  if  used  appropriately  definitely  on  slower 
predictable  programs,  maybe  not  on  dynamic 
programs 

15 

Software  metrics,  schedule,  IMS,  EVM,  a-To 
determine  of  on  track  (cost  and  schedule),  b-  Bi¬ 
weekly  to  monthly  it  depends  on  the  metric,  c- 
Not  necessarily,  have  to  accomplish  enough 
(between  receipt  of  metric), 

Memory  through  put  and  EVM  are  the  biggest 
ones 

A.  Monitor  program 

B.  EVM  monthly 

C.  No 

D.  No,  cost  and  schedule  already  get  monthly 

16 

We  get  them 

Requirements,  Software  Trouble  Reports, 
Prioritizations,  Integrated  Master  Schedule, 
People  (enough  to  do  the  job?) 

A.  Track  and  predict  health  of  effort 

B.  Weekly 

C.  No 

17 

A.  Management  used  them  more 

B.  Some  policy  makes  recommendations 

D.  Yes  it  would  benefit  them  but  hard  to 
determine  measurements 

Use  DCMA  along  with  program  office  people  to 
evaluate  how  contractor  is  doing 

D.  PM  probably  gets  them,  I  don’t  see  them  but 
some  of  my  engineers  probably  see  them 

18 

N/A 

D.  Yeah  closer  of  open  item 

There  is  difficulty  in  getting  and  applying 
metrics. . .  .not  very  efficient  and  not  used 
effectively 

A.  It  might  help  if  there  was  an  agreement  on 
how  metrics  were  going  to  be  used 

D.  Yes  it  would  be  a  benefit  to  get  those 

19 

EVM 

A.  Whole  group  monitors 

B.  Monthly 

C.  Yes  often  enough 

D.  Yes  its  beneficial 

Defect,  SLOC,  memory  through  put 

A.  Manage  progress  and  look  for  concerns 

B.  All  the  time 

C.  No  but  keep  on  top  of  them 

D.  Get  them  already 

20 

Cost,  Schedule,  content  data 

A.  To  determine  future  costs,  evaluate 
performance,  cost 

B.  Monthly 

C.  No 

D.  We  do  get  this 
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Question  28 


Yes,  it  was 

11 

Yeah 

Don’t  see  it 

12 

Perhaps  some  standard  metrics,  but  each 
programs  have  their  own  vocabulary  so  maybe 
not  really  standard 

Yes 

13 

Have  metrics  specific  to  each  phase  of  the 
program,  for  example  test  metrics.  Yes  very 
beneficial  to  predict  or  detect  deficiency 

We  really  don’t  do  development...  just  do  mods 

14 

Didn’t  deal  with  it,  All  ready  established 
programs,  Yes  familiar  with  it,  most  PMs  are 
probably  not 

Yes,  but  the  contractor  must  be  able  to  do  it 
without  getting  in  the  way  of  development 

15 

Don’t  think  you  can  develop  specific  standard 
metrics,  Yeah,  but  don’t  know  how  you  would 
determine  it 

I  think  there  was  policy  suggesting  this,  most 
people  are  doing  this  anyways  with  CMM  level  3 

16 

Could  be  a  list  and  you  pick 

Shy  away  from  standard  metrics. .  .otherwise  you 
get  metrics  for  the  sake  of  getting  metrics 

17 

Theoretically  yes,  but  how  do  you  measure? 

I  think  so 

B.  Yes  I  herd  of  it 

18 

Yes  they  would  be 

Number  of  defects  remaining  would  be  beneficial 

19 

Use  AF  standard  metrics  as  required  and  policy 

I  would  say  no,  not  easily  measured,  not  useful 

20 

Yes  they  are 
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Question  29 


1 

Meeting  requirements 

11 

Not  driving  schedule  impacts  and  are  they  able  to 
integrate  our  products 

2 

Turn  it  on  it  works,  doesn’t  have  issue  later 

12 

Cost,  Schedule,  Performance  A.  All  ready  graded 
on  these  (Cost,  Schedule,  and  Performance) 

3 

How  delighted  the  user  is.  Not  a  quantitative 

measure 

13 

Flight  test  and  integration  for  program  as  a  whole 
for  software  only  measure  by  metrics 

1 

Cost  and  schedule 

14 

Don’t  get  cancelled,  Tough  to  standardize 

5 

On  key  events,  for  examble  first  flight  test 

15 

Deliver  on  time  and  budget  along  with  user 
satisfaction,  Cost,  schedule  and  performance  are 
met  (TPNs  and  KPPs  are  met 

6 

EVM 

16 

Was  I  able  to  take  people's  money  and  look  at 
EVM,  tech  review,  and  schedule 

1 

You  do  what  you  say  you  are  going  to  do 

17 

Something  that  meets  requirements 

8 

In  the  test  bed,  the  program  does  what  it  is 
supposed  to  do  and  also  by  schedule 

18 

Meeting  delivery  date  and  do  they  have  any 
outstanding  items 

9 

Quality  of  products,  maintaining  cost  and 
schedule 

19 

When  you  fly  does  it  work  as  its  supposed  to 

10 

Achive  our  schedule 

20 

Wing,  ACC,  customer  feedback 
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Question  30 


1 

Yes,  has  been  very  helpful.  Can  track  where  they 

are 

11 

Depends  on  size  of  program 

2 

Yes,  it  would  be  nice 

12 

Not  necessarily-  if  firm  fixed  price  don't  need  it 

3 

Yes  they  should  and  we  do 

13 

Yes  we  have  it. .  .have  monthly  EVM  report 

1 

Yes  I  like  EVM  and  it  has  been  pretty  successful 

14 

Depends  on  the  contract  vehicle-  if  you  have  a 
firm  fixed  price  can't  really  measure  it 

5 

Yes,  should  but  not  if  contractor  cannot  support 
it 

15 

Yes,  it  measures  ability,  but  doesn't  address  if 
you  are  getting  for  money  being  spent 

6 

Yes,  have  to  make  sure  they  are  planning  at  that 
level 

16 

Most  contactors  have  an  implemented  EVM 
system 

1 

Y eah,  but  more  for  management  not  engineering 

17 

Yes 

8 

Use  it  religiously  -used  to  determine  award  fee 

18 

Yes 

9 

Y eah  defiantly 

19 

It  depends  on  what  your  doing  and  magnitude- 
small  project  may  not  benefit 

10 

Yeah  we  do 

20 

Not  under  PBL,  not  in  our  case 
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Question  3 1 


For  the  program?  Yes  previous  job  was  the  same 

11 

Y eah  but  we  don't  have  big  software  component 
right  now 

Yes,  got  more  than  we  need  in  the  program  office 

12 

No,  Engineers  that  understand  Software  ,  PM 
that  really  understands  it,  needs  depth  (a  good 

PM  that  gets  the  process),  Some  people  have  to 
work  harder. .  .get  over  worked  or  buy  ANAS 
support,  People  are  over  worked 

No  we  don’t 

13 

Right  skills  yes,  could  use  more  though,  Need  to 
work  hand  in  hand  with  developer  and  need 
competent  software  engineer 

They  have  got  way  too  much 

14 

Had  no  resources. .  .but  across  the  board  no,  we 
hire  the  wrong  engineers,  Nave  a  back  ground  in 
software  development,  maybe  a  computer 
science  engineer 

No 

15 

We  do  now,  just  ramped  up  the  number  of  people 

Yes 

16 

We  do  here  in  the  program  office,  debatable 
whether  the  contractors  have  enough 

No 

17 

Yes,  but  cutting  a  lot  of  personnel 

Yes 

A.  Electrical  Engineers  and  some  Aero  Engineers 

18 

No,  need  more  management  and  software 
management  have  too  many  coders 

Yes,  but  at  one  time  it  was  not  true  in  a  certain 
area  of  development  didn’t  have  enough 
personnel,  this  resulted  in  a  schedule  slip 

A.  Need  specific  language,  Overtime  there  has 
been  limited  people  and  there  is  only  so  much 
you  can  do,  Need  people  experienced  with 
interface  work  or  schedule  slips 

19 

No  high  level  and  low  level  (program  level  or 
lower) 

Yes 

20 

Yes 
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Question  32 


1 

Never  gone  outside  of  Wing 

11 

You  relay  on  center  and  contract  out 

2 

We  don’t  have  any  SPO  expertise,  most  program 
office  people  are  managers 

12 

N/A 

3 

Don’t  have  enough.  Everyone  is  short  on  people 
not  just  software.  Had  to  use  center  and 
contractor  in  the  past 

13 

Work  with  contractor  as  a  team  and  have  good 
communication 

B 

We  have  a  couple  of  key  people.  EN  will 
backfill 

14 

Do  a  lot  of  contracting  out,  TITAN 

5 

No 

15 

Yes  from  software  perspective 

6 

No 

16 

Got  enough  in-house  for  management 

B 

No  have  to  contract  out  for  software  expertise 

17 

Overall  running  low  on  software  expertise 

8 

No  home  office  support 

18 

No,  contracting  it  out 

9 

At  certain  times  we  rely  on  engineering  staff  as  a 
supplement,  but  not  day  to  day  activities  . .  .we 
have  enough  and  we  do  some  contracting  out 

19 

Have  1  govt  and  1  contractor,  yes  you  have  to 
contract  out 

10 

We  do  not  with  the  SPO  and  don't  contract  out 

either 

20 

No,  rely  on  organic  software  expertise 
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Question  33 


We  really  don’t  do  in-house  construction 

11 

Don't  really  do  software  in-house,  contractor 
should  do  the  work  we  should  set  requirements 
and  manage 

Yes,  reduce  it  or  get  out  of  the  business 

12 

As  much  as  possible,  don’t  need  to  do  in-house 
coding 

Don’t  know 

13 

Don’t  think  so,  especially  with  more  and  more 
software  intensive  programs, 

They  should  not  be  developing  software,  they 
should  only  be  fixing.  There  are  benefits  to 
having  contractor  doing  it 

14 

Pretty  much  done  only  at  Gunter  (Maxwell  AFB) 

Depends,  can  see  good  both  ways 

15 

Yes...  put  govt,  in  critical  path 

No,  ALC  do  software  development,  do  it  well 
and  cheaply,  good  or  better  report  w/user 

16 

No  if  you  have  done  software  the  better  you  are 
at  managing  software 

Don’t  do  in-house,  ALCs  do  this  type  of  stuff 

17 

EN  policy  doesn't  allow  for  organic  support,  not 
sure  anybody  does  in-house 

Very  expensive,  can’t  get  anymore  of  these 
people  because  they  were  capped  by  regulation 

18 

Y eah  they  should,  we  are  contracting  it  out 

No  should  not  reduce,  keep  at  current  level 

19 

All  ready  happened,  being  forced  to  go  to 
contractors 

We  don't  do  a  lot  of  coding,  Navy  is  more 
organic. .  .our  contractor  will  continue  to  have 
maintenance  responsibility 

20 

No 
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Question  34 


Y es  to  both 

A.  Yes 

B.  Had  a  separate  group  that  just  did  a  V&V 

11 

Don't  do  these 

None 

A.  No 

B.  It  would 

12 

Try  to  do  it  in-house  B.  Beg  borrow  and  steal  for 
additional  resources 

None  in-house.  Too  much  software  code  to  do  in- 

house 

A. N/A 

B.  It  would 

13 

Did  a  lot  in-house,  lots  of  document  review 

A.  Yes 

B.  Yes 

Partial  coding,  normal  PM  overhead  activities 

14 

Done  in  house,  (Government)  contractor 
oversees  it 

None 

A.  It  would  be 

B.  It  would  be 

15 

Do  Design  and  V&V 

A.  Yes 

B.  No. ..going  to  staff  appropriately 

Do  in-house,  don’t  do  V  &  V,  don’t  do 

A.  Design  and  reviews,  yes 

B.  No 

16 

We  participated,  had  insight,  influence,  and 
control 

A.  Very  beneficial  to  have  influence  and  control 

B.  Yeah 

CMM-  informally,  Design/Code  reviews-  Yes 

A.  Yes 

B.  Yes  brought  in  help. 

17 

Yes 

A.  Yes  there  is  benefit  and  encourage  it 

B.  Yes 

Don’t  do  V&V  anymore. .  .rely  in  contractor  to 
do  their  thing  and  send  our  engineers  for 
oversight 

18 

Not  sure 

Design/code  reviews,  and  V&V  were  done  in- 
house,  we  also  did  code  sampling 

A.  Yes  it  was  beneficial 

B.  Yes  at  certain  times  it  did  require  additional 
people 

19 

CMM-  No  Design/Code  Reviews-  Kind  of,  V&V 
have  in-house  tester 

A.  Would  not 

B.  Yes  it  would  and  that’s  why  don't  we  do  it 

Do  CMM  &  design/code  reviews...  don't  do 

V&V  but  have  oversite 

A. N/A 

B.  No 

20 

Code  reviews  and  V&V 

A.  Yes 

B.  Yes 
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Question  35 


Yes  it  would  be  beneficial  to  stay  with  the 
program  through  critical  phase.  No...  turn  over 
during  critical  phase  though 

11 

Yeah,  3  yrs  is  about  right 

No,  not  any  more.  It  heps  though,  but  can  also 
become  more  complacent.  Good  for  people  to 
get  good  experience 

12 

Need  to  show  growth,  so  changing  jobs  a  good 
thing  for  the  military  folks,  civilian  folks  can  stay 
with  the  program  longer 

Absolutely 

13 

Yes,  I  think  so. .  .need  experience  people  that 
know  every  nut  and  bolt 

Yes,  the  longer  you  are  around  the  smarted  you 
are,  but  threre  comes  a  point  when  its  time  to 
leave 

14 

Depends  on  the  person 

Don’t  know,  depends  on  who  you  are 

15 

Continuity  it  is  very  important...  leaving  after  1 
or  2  years  is  not  good 

Beneficial  to  the  program,  but  not  to  the  person 

16 

Optimal  is  3-4  years...  enough  for  continuity  and 
then  should  move  to  broaden  their  view 

Yes,  5  years  is  probably  long  enough  though 

17 

Yes 

Should  move  to  get  new  perspectives 

18 

Yes  good  for  any  program,  relatively  long 
enough  but  not  to  long 

Yes  it  is  defiantly  beneficial 

19 

Defiantly,  loss  of  institutional  knowledge 

It  is  always  benifical  to  keep  corporate 
knolwedge  there 

20 

Yes 
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Question  36 


Don’t  really  know  about  that 

11 

It  would  be,  gets  other  sides  point  of  view...  a 
simple  requirement  change  on  one  side  is  not  so 
simple  on  the  other 

It  could  be  good  or  bad,  depends  on  the  people 

12 

Would  be  very  inefficient,  different  cultures  and 
would  take  a  while  to  catch  on,  EWI-  is  a 
fantastic  program 

I  think  so,  worked  in  the  past 

13 

Cost  involved  . .  .can’t  afford  it 

Yeah,  it  wouldn’t  hurt.  But  can  end  up  an  outcast 

14 

Tough  to  do. . .  contractor  personnel  even  tougher 
to  do. .  .can  sit  in  IPT,  but  not  into  an 
organization  (no  real  benefit  to  it) 

It  would  be  counter  productive,  in  a  large 
organization  maybe 

15 

No  not  for  all  software  development,  need  good 
communication 

Probably  not  a  whole  lot,  we  both  understand 
how  each  side  works,  EWI 

16 

Don't  think  so,  too  much  overhead,  not  past 
requirement  phase 

Yes,  but  in  small  doses. .  .maybe  2  weeks  at  a 
time 

17 

No,  must  be  familiar  with  each  side  of  process 

Yes  it  would  a  little 

A.  EWI  (Education  with  Industry) 

18 

It  would  be  beneficial,  but  not  necessary 

No  real  significant  benefit,  we  are  all  doing  a  lot 
of  different  things  so  it  may  be  hard  to  do 

19 

It  would  be  beneficial,  but  not  sure  if  you  really 
can  do  it 

Yeah,  I  would  say  so 

20 

Yes  in  some  cases 
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Question  37 


Very  important,  its  critical,  need  them  from  day 
one  till  fielding 

11 

Extremely  important  especially  in  requirement 
definition  and  changing  requirements 

A.  They  are  pretty  involved 

Its  good  to  get  the  users  involved 

12 

Absolutely  critical 

Incredibly  important 

A.  Not  as  involved  as  should  be 

13 

Very  important.  Lesson  learned  and  it’s  proven. 
Need  to  keep  in  requirements  definition 

You  help  get  them  involved 

14 

Involved  early  on 

A.  Drove  a  lot  of  things  they  shouldn’t 
have. .  ..were  involved  in  early  meetings,  - 
Personality  driven,  -  need  to  find  were  it  (user 
involvement)  fits  into  your  program 

Yes,  absolutely  important 

A.  Involvement  increasing 

15 

Defiantly  important,  ensure  requirements  meet 
user  expectations  and  understanding  A. 

Participate  all  reviews 

User  provides  requirements  in  priority  and  input, 
estimate  concept  of  operations.  Without  user 
there  would  be  more  problems 

A.  Formal  meetings  and  a  lot  of  informal 
exchanges 

16 

Extremely  critical...  need  to  get  their 
expectations  and  their  understanding  of 
requirements 

A.  Not  as  involved  as  we  would  like  them  to  be 

Very  important 

A.  All  major  reviews 

17 

Always  have  user  involved,  they  develop  the 
requirements 

A.  They  are  involved 

Very  important 

A.  Very  Involved 

18 

Very  important,  have  to  be  involved  up  front  for 
requirements  definition 

Defiantly  important 

A.  They  should  have  been  more  involved,  they 
were  only  moderately  involved 

19 

Vital  early  and  continuously  (ACC  is  intimately 
involved) 

Extremely  important 

A.  They  are  very  involved 

20 

Critical 

A.  Adequately. .  ..determined  by  training  and 
mission 

121 


Question  38 


Used  a  matrix  that  defines  all  requirements  and  it 
defined  DT&E  plus  requirements  tracable  all  the 
way  back  to  ORD. 

11 

N/A 

You  never  do 

12 

Exercise  all  required  functionality,  all  interfaces, 
load  test,  and  make  sure  data  is  going  through 
correct,  Killing  yourself  by  doing  that 

Don’t  know 

13 

If  we  have  a  successful  FOQ&T.  and  satisfied 
test  cases,  If  we  have  a  successful  FOQ&T.  and 
satisfied  test  cases 

Rely  on  test  community.  Great  test  plan  and  Test 
Readiness  Review 

14 

No  answer  -  didn’t  get  that  far 

Requirements  are  met,  user  is  okay  with  it 

15 

Have  requirements  correlation  matrix...  as  long 
as  every  requirements  can  be  tested 

No  specific  criteria 

16 

Have  consensus  on  the  govt,  team,  get  ITT 
together  and  look  at  test  cases 

Flight  test 

17 

Experience 

1.  Ground  test,  2.  flight  test,  get  user 
participation  in  both.  The  sooner  the  user  is 
involved 

A.  It  would  be  ideal,  but  not  enough  time, 
money,  or  people 

18 

Requirements  traceability,  if  requirements  can  be 
shown  they  were  tested 

When  you  know  you  requirements  are  thoroughly 
checked  out,  Contractor  should  exercise  each 
module  extensively,  but  in  DT&E  not  realistic  to 
test  every  instruction 

19 

Don't  know  depends  on  application 

Trial  and  error,  comes  through  experience  and 
knowledge  of  the  system 

20 

Minimum  problems  that  make  it  to  the  field 
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Question  39 


Not  so  sure,  I  need  to  have  some  operational 
testing  to  be  fieldable 

11 

Risk  management  decision 

Yes  if  the  customers  want  it 

12 

No,  if  you  do  that  you  have  to  be  sure  you  are 
getting  the  right  data  out,  really  need  thorough 
tests 

Only  time  you  want  to  do  that  is  if  test  guys  are 
constained  to  production 

13 

It  depends,  if  software  that  could  affect  life  then 
yes,  but  if  not  mission  critical  then  yes 

N/A 

14 

No  answer  -  didn't  get  that  far 

Absolutely 

15 

No  OT  is  a  different  way  of  looking  at  it,  could 
be  sometimes...  if  it  transparent  to  users 

Disaster  waiting  to  happen.  Small  limited  cases, 
but  in  general  no 

16 

Y eah  if  user  has  reviews  it,  but  probably  not  a 
good  idea 

No 

17 

No...  not  safety  critical  stuff,  but  in  some  cases 
yes 

Yes  and  No  depends  on  level  of  testing 

18 

Yes  provided  its  been  given  some  kind  of  OA 

Don’t  see  a  problem  if  there  is  a  pressing  need 
and  risks  are  understood 

19 

Depends  on  application  -  Business  systems  yes  - 
embedded  systems  no 

No 

20 

Depends  on  software 
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Question  40 


Yes,  the  maintainers  are  the  contrators.  Yes  they 
were  involved 

11 

Don't  know 

No,  in  some  cases  they  bring  in,  but  sometimes 
not.  It  would  be  good  to  though 

12 

Software  goes  back  to  the  contractor,  so  not 
really  relevant  here 

Future  maintainers  are  doing  development 

13 

N/A 

Yes-  (Base  Masked) 

14 

No  answer  -  didn't  get  that  far 

No 

15 

No 

No 

16 

No,  they  are  off  doing  their  own  software 
maintenance  and  don't  have  time  or  people 

Yes  on  this  program,  my  previous  program  no 

17 

Not  done  very  much 

On  F-16  they  were,  but  here  not  so  much 

18 

Not  sure 

Yes  they  were  the  same  one  who  developed  the 
software 

19 

Not  sure 

No 

20 

Yes 
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Question  4 1 


Locations  Masked 

A.  Yes  they  have  their  own 

11 

Doesn't  apply  but  kind  of  by  implementing  on 
aircraft 

A.  Yes 

A.  Yes 

12 

AFOTEC  delegated  it  to  (masked)  A.  Yes  they 
were  provided 

AFOTEC 

A.  Yes 

13 

Main  operating  base  (masked)  A.  Yes 

Doing  FDE  at  (masked) 

A.  Yes 

14 

No  answer  -  didn’t  get  that  far 

AFOTEC 

A.  Yes 

15 

Command  squadron 

A.  Yes 

OT  &  E,  AFOTEC 

A.  Yes 

16 

(Masked) 

A.  Yes 

Customer/ AFOTEC 

A.  Yes 

17 

Flight  controls. . ..  (Masked) 

A.  Yes 

AFOTEC 

18 

User  conducts  OT&E,  Customer  Acceptance 
Testing 

A.  As  required 

AFOTEC 

A.  There  was  no  real  need  for  a  dedicated 
facility,  but  arrangements  were  made 

19 

AFOTEC 

A.  Yes 

Intergrated  test  force 

A.  Yes 

20 

(Masked) 
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Question  42 


1 

Yes,  we  try  to  have  them  involved  from  the 
beginning 

11 

Yes  absolutely  -  need  flexible  software 

2 

Yes,  we  try  to  have  them  involved  from  the 
beginning 

12 

Yes,  plan  for  it  in  your  Acquisition  Strategy 

3 

Yes 

13 

Yes 

B 

Yes  we  are  trying  to  do  that  with  this  suite 

14 

No  answer  -  didn’t  get  that  far 

5 

Absolutely 

15 

Yes  have  to 

6 

Not  given  a  high  enough  priority  early  in 
development 

16 

Depends  if  you  have  a  robust  well  documented 
software  architecture 

B 

Yes 

17 

Yes 

8 

Defiantly 

18 

Yes  if  you  have  a  plan  to  resolve  them 

9 

They  are  addressed  early  and  sometimes  software 
maintenance  issues  are  forced  on  you.  They 
decision  you  make  early  maybe  arbitrarily 
reversed  later  though 

19 

Yes  its  an  area  often  over  looked 

10 

Yes,  we  deal  with  the  maintenacne  issuse  in  our 
releases 

20 

Yes 
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Question  43 


Contractor  who  maintains  it..  It's  NDI,  buying 
software,  not  developing  it 

11 

Contractor 

Contractor 

12 

Contractor,  A.  Acq.  Strategy 

Contractor  for  now,  but  want  to  move  to  depot 

13 

Prime  contractor 

A.  Don't  evaluate  it  anymore,  contractor 
maintained 

Contractor,  we  usually  don’t  do  maintained,  we 
just  wait  till  next  release,  unless  its  a  critical 
failure 

14 

No  answer  -  didn’t  get  that  far 

Contractor 

15 

Contractor,  we  don't  do  organics...  its  only  CLS 

(Masked) 

A.  Source  of  repair  analysis  detennined 
(masked)  cheaper  than  the  contractor. ... 
extensive  study 

16 

Contractor,  we  can't  hire  the  people  to  do  it  in- 
house 

Contractor 

A.  With  a  complicated  system  lean  toward 
contractor. .  .they  have  insight 

17 

Contractor  ,  we  can't  do  in  house  anymore. . .  .not 
feasible...  don't  have  enough  people 

CLS  -  Contractor  Logistic  Support 

18 

Contractor 

Contractor  maintained 

A.  Does  the  government  have  the  facilities  and 
manpower?  It  may  come  down  to  a  cost  trade¬ 
off. 

19 

Weapon  system  support  center  in  (masked),  both 
govt  and  contractor  do  to  security  reasons 

Contractor 

A.  Don't  have  a  choice  use  contractor  support, 
but  you  could  do  a  cost  befit  analysis 

20 

Contractors  and  organic  engineers 

A.  Based  on  product 
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Question  44 


Contractor  maintanence  plan 

11 

Preliminary,  but  now  on  front  end  of 
development  and  they  are  scrambling 

No  comment 

12 

Contractor  will  do  plan 

Yes 

13 

Yes,  open  software  every  2  years  to  do 
maintenance 

Yes,  normal  DR  process 

14 

No  answer  -  didn’t  get  that  far 

There  is  a  process  for  it 

15 

Use  same  plan  that  exists 

No,  might  have  one  but  very  old  and  not  used, 
had  a  transition  plan 

16 

Not  really...  Software  maintenance  planning  has 
been  ad-hoc 

Yes 

17 

Normally  included  in  software  development  plan 

Yes 

18 

Most  contracts  have  some  kind  of  sustainment 
planning,  built  in  as  a  deliverable  ,  but  govt 
didn’t  come  up  with  plan 

It  wasn’t  a  detailed  plan,  since  it  was  going  to  be 
maintained  by  the  contractor 

19 

Yes  its  been  followed  for  the  last  20  years 

No  it  was  not 

20 

Yes 
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Question  45 


Yes,  OSS&E  process 

A.  We  have  a  small  cycle  time  pretty  quick,  no  2 
yr  long  cycle 

11 

Based  on  contractor  processes 

Yes 

A.  Yes 

12 

Absolutely  designed  process,  Also  depends  on 
decision  authority  for  reviews.  A-  On  small 
programs,  on  large  ones  can  have  trouble  with 
scheduling  PEO  review 

Yes 

A.  Hard  to  say  but  know  what  we  want  to  do 

13 

N/A 

Use  suites 

14 

No  answer  -  didn’t  get  that  far 

We  are  getting  the  process  to  mature 

A.  No  it  is  counter  productive  coming  of  an 
ACTD...  it’s  a  learning  curve. 

15 

Yes  defiantly  sustainment  block  process 

A.  No  doesn't  have  anything  to  do  with  software 
development 

Yes 

A.  Have  modified  process  to  reduce  time  to  get 
fielded 

16 

Have  a  sustainment  block  process,  but  not  on 
development  side 

Yes,  that’s  how  we  do  it 

A.  Yes 

17 

Its  in  life-cycle,  spirals 

A.  You  can  spend  more  time  doing  models...  it 
can  but  tend  not  to 

Yes 

A.  No,  vary  from  contractor  to  contractor 

18 

IT  lean  process  EA  and  Spiral 

A.  Yes 

Yes  and  No,  tailored  to  the  organization 

19 

Block  increments,  very  formal  process 

A.  was  not  important  to  do  things  quickly,  but 
you  have  to  do  it 

Yes,  block  release,  flight  test  updates 

A.  Yes  it  has 

20 

Yes 

A.  Yes 

Question  46 


Have  a  DR  (deficiency  report),  standard  AF 
program 

11 

Based  on  contractor  processes 

Yes 

12 

Absolutely-  also  grade  severity,  Varies  by 
program 

Yes 

13 

Yes,  contractor  has  deficiency  report  and 
procedures 

Yes  we  do,  contractor  tracks  all  problems 

14 

No  answer  -  didn’t  get  that  far 

Yes,  the  SOW  system  process,  MIP  review  board 
handles  DRs. 

15 

Absolutely  -  defined  by  discrepancy  reports 

Yes 

A.  A  few  processes. ..DR  and  WIT 

16 

Not  sure  what  the  contractor  process  is 

Yes,  software  trouble  reports  and  anyone  can 
write  one  including  the  user 

17 

Most  programs  do,  whether  or  not  is  a  different 
thing 

Yes  we  use  our  own  process 

18 

Yes,  but  not  in  software  processes 

There  was  a  formal  process 

19 

Yes  very  formal  process  occurring  at  all  levels 

Yeah  system  problem  reports,  problem  reporting 
system 

20 

Yes 
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